Jul 26 16:06:46.028028 ip-10-0-130-70 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 26 16:06:46.028041 ip-10-0-130-70 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 26 16:06:46.028051 ip-10-0-130-70 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 26 16:06:46.028365 ip-10-0-130-70 systemd[1]: Failed to start Kubernetes Kubelet. Jul 26 16:06:56.187541 ip-10-0-130-70 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 26 16:06:56.187557 ip-10-0-130-70 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 8dc14d3dd8bc414199fa1fc48878d4b2 -- Jul 26 16:08:46.307515 ip-10-0-130-70 systemd[1]: Starting Kubernetes Kubelet... Jul 26 16:08:46.668711 ip-10-0-130-70 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 26 16:08:46.668711 ip-10-0-130-70 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 26 16:08:46.668711 ip-10-0-130-70 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 26 16:08:46.668711 ip-10-0-130-70 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 26 16:08:46.668711 ip-10-0-130-70 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 26 16:08:46.669398 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.668762 2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 26 16:08:46.672569 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672551 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 16:08:46.672569 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672567 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672572 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672575 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672579 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672582 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672584 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672587 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672590 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672593 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672595 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672598 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672601 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672603 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672606 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672609 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672612 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672615 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672618 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672620 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672623 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 16:08:46.672635 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672625 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672628 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672631 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672634 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672637 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672640 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672643 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672646 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672649 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672652 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672655 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672657 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672659 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672662 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672665 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672668 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672670 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672673 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672675 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672680 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 16:08:46.673146 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672684 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672687 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672690 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672693 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672695 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672698 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672700 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672703 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672705 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672708 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672710 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672713 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672716 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672719 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672722 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672725 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672728 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672731 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672734 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672736 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 26 16:08:46.673636 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672739 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672742 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672744 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672747 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672750 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672752 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672755 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672758 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672762 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672765 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672769 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672772 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672775 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672778 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672780 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672783 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672785 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672788 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672790 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 16:08:46.674134 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672793 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672795 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672798 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672800 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672803 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672805 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672808 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672811 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672814 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672817 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672819 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672822 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672824 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672827 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672830 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672834 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672837 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672840 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.672842 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 16:08:46.674608 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673619 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673626 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673630 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673633 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673636 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673641 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673644 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673647 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673650 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673653 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673656 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673659 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673662 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673664 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673667 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673670 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673672 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673675 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673677 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 16:08:46.675059 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673680 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673684 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673687 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673689 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673692 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673695 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673698 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673700 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673702 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673706 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673708 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673711 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673713 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673716 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673719 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673723 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673726 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673728 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673731 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673734 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 16:08:46.675534 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673737 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673740 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673742 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673745 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673747 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673750 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673753 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673756 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673758 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673761 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673763 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673766 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673769 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673773 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673775 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673778 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673780 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673783 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673786 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673788 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 16:08:46.676019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673791 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673793 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673796 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673798 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673801 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673803 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673806 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673808 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673811 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673815 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673818 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673820 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673823 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673826 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673828 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673831 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673833 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673836 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673839 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 16:08:46.676526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673843 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673846 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673849 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673851 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673854 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673857 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673860 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673864 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673867 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673870 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673873 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673876 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673878 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673881 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673884 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673888 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673891 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673894 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673897 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 16:08:46.677015 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673900 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.673903 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674345 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674351 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674355 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674358 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674361 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674364 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674366 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674369 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674372 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674375 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674377 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674380 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674382 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674385 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674388 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674390 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674393 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674396 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 16:08:46.677498 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674399 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674402 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674405 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674407 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674410 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674412 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674415 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674417 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674420 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674422 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674425 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674427 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674430 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674432 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674442 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674444 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674447 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674450 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674452 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674455 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 16:08:46.677979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674459 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674464 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674467 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674470 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674473 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674476 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674479 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674482 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674484 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674487 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674490 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674492 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674495 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674498 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674500 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674503 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674505 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674508 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674510 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674513 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 16:08:46.678547 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674516 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674518 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674521 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674523 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674525 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674528 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674533 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674535 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674537 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674540 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674542 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674545 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674556 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674559 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674561 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674564 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674566 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674569 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674572 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674574 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 16:08:46.679023 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674577 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 16:08:46.679527 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674581 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 16:08:46.679527 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674584 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 16:08:46.679527 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674587 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 16:08:46.679527 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674590 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 16:08:46.679527 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674593 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 16:08:46.679527 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674595 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 16:08:46.679527 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674598 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 16:08:46.679527 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674602 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 16:08:46.679527 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674604 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 16:08:46.679527 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674607 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 16:08:46.679527 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674610 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 16:08:46.679527 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674612 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 16:08:46.679527 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674615 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 16:08:46.679527 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674618 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 16:08:46.679527 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674620 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 16:08:46.679527 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674623 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 16:08:46.679527 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674626 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 16:08:46.679527 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674630 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674632 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.674635 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674714 2563 flags.go:64] FLAG: --address="0.0.0.0" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674722 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674728 2563 flags.go:64] FLAG: --anonymous-auth="true" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674732 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674747 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674751 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674755 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674759 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674764 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674767 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674770 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674774 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674777 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674780 2563 flags.go:64] FLAG: --cgroup-root="" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674782 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674786 2563 flags.go:64] FLAG: --client-ca-file="" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674789 2563 flags.go:64] FLAG: --cloud-provider="external" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674792 2563 flags.go:64] FLAG: --cluster-dns="[]" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674797 2563 flags.go:64] FLAG: --cluster-domain="" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674799 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674803 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 26 16:08:46.679986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674806 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674809 2563 flags.go:64] FLAG: --container-log-max-files="5" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674813 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674816 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674819 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674822 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674826 2563 flags.go:64] FLAG: --contention-profiling="false" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674828 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674832 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674835 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674838 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674842 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674845 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674848 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674851 2563 flags.go:64] FLAG: --enable-load-reader="false" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674854 2563 flags.go:64] FLAG: --enable-server="true" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674857 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674862 2563 flags.go:64] FLAG: --event-burst="100" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674865 2563 flags.go:64] FLAG: --event-qps="50" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674870 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674873 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674876 2563 flags.go:64] FLAG: --eviction-hard="" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674880 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674883 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674886 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 26 16:08:46.680591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674888 2563 flags.go:64] FLAG: --eviction-soft="" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674891 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674894 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674897 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674900 2563 flags.go:64] FLAG: --experimental-mounter-path="" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674903 2563 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674906 2563 flags.go:64] FLAG: --fail-swap-on="true" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674909 2563 flags.go:64] FLAG: --feature-gates="" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674913 2563 flags.go:64] FLAG: --file-check-frequency="20s" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674916 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674919 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674922 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674925 2563 flags.go:64] FLAG: --healthz-port="10248" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674928 2563 flags.go:64] FLAG: --help="false" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674931 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-130-70.ec2.internal" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674935 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674938 2563 flags.go:64] FLAG: --http-check-frequency="20s" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674941 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674945 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674948 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674951 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674953 2563 flags.go:64] FLAG: --image-service-endpoint="" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674956 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 26 16:08:46.681194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674959 2563 flags.go:64] FLAG: --kube-api-burst="100" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674963 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674967 2563 flags.go:64] FLAG: --kube-api-qps="50" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674971 2563 flags.go:64] FLAG: --kube-reserved="" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674974 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674977 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674980 2563 flags.go:64] FLAG: --kubelet-cgroups="" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674983 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674985 2563 flags.go:64] FLAG: --lock-file="" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674988 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674991 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.674994 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675000 2563 flags.go:64] FLAG: --log-json-split-stream="false" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675002 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675005 2563 flags.go:64] FLAG: --log-text-split-stream="false" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675008 2563 flags.go:64] FLAG: --logging-format="text" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675011 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675014 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675017 2563 flags.go:64] FLAG: --manifest-url="" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675020 2563 flags.go:64] FLAG: --manifest-url-header="" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675024 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675027 2563 flags.go:64] FLAG: --max-open-files="1000000" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675031 2563 flags.go:64] FLAG: --max-pods="110" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675034 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675039 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 26 16:08:46.681790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675041 2563 flags.go:64] FLAG: --memory-manager-policy="None" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675045 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675048 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675051 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675053 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675061 2563 flags.go:64] FLAG: --node-status-max-images="50" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675064 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675067 2563 flags.go:64] FLAG: --oom-score-adj="-999" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675070 2563 flags.go:64] FLAG: --pod-cidr="" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675074 2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675079 2563 flags.go:64] FLAG: --pod-manifest-path="" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675084 2563 flags.go:64] FLAG: --pod-max-pids="-1" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675087 2563 flags.go:64] FLAG: --pods-per-core="0" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675090 2563 flags.go:64] FLAG: --port="10250" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675093 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675096 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b05ebe650580cdfa" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675100 2563 flags.go:64] FLAG: --qos-reserved="" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675102 2563 flags.go:64] FLAG: --read-only-port="10255" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675105 2563 flags.go:64] FLAG: --register-node="true" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675108 2563 flags.go:64] FLAG: --register-with-taints="" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675112 2563 flags.go:64] FLAG: --registry-burst="10" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675115 2563 flags.go:64] FLAG: --registry-qps="5" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675117 2563 flags.go:64] FLAG: --reserved-cpus="" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675120 2563 flags.go:64] FLAG: --reserved-memory="" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675124 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 26 16:08:46.682597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675127 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675130 2563 flags.go:64] FLAG: --rotate-certificates="false" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675133 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675135 2563 flags.go:64] FLAG: --runonce="false" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675138 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675141 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675145 2563 flags.go:64] FLAG: --seccomp-default="false" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675148 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675150 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675153 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675156 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675159 2563 flags.go:64] FLAG: --storage-driver-password="root" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675162 2563 flags.go:64] FLAG: --storage-driver-secure="false" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675165 2563 flags.go:64] FLAG: --storage-driver-table="stats" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675167 2563 flags.go:64] FLAG: --storage-driver-user="root" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675170 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675173 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675177 2563 flags.go:64] FLAG: --system-cgroups="" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675181 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675186 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675189 2563 flags.go:64] FLAG: --tls-cert-file="" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675192 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675197 2563 flags.go:64] FLAG: --tls-min-version="" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675199 2563 flags.go:64] FLAG: --tls-private-key-file="" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675202 2563 flags.go:64] FLAG: --topology-manager-policy="none" Jul 26 16:08:46.683193 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675205 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675208 2563 flags.go:64] FLAG: --topology-manager-scope="container" Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675211 2563 flags.go:64] FLAG: --v="2" Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675215 2563 flags.go:64] FLAG: --version="false" Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675232 2563 flags.go:64] FLAG: --vmodule="" Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675235 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.675239 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675364 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675367 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675370 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675373 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675376 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675378 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675382 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675384 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675387 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675389 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675392 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675394 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675397 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675399 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675401 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 16:08:46.683844 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675404 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675406 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675409 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675413 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675417 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675419 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675422 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675425 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675427 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675430 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675433 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675435 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675438 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675441 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675443 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675446 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675448 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675451 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675453 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675456 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 16:08:46.684399 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675459 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675461 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675464 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675466 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675469 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675472 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675474 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675477 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675479 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675482 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675484 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675487 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675490 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675492 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675494 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675498 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675501 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675504 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675506 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675509 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 16:08:46.684890 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675511 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675514 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675517 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675519 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675522 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675524 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675527 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675529 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675531 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675534 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675536 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675540 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675544 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675547 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675550 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675552 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675555 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675557 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675560 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 16:08:46.685384 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675562 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675565 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675568 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675570 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675573 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675576 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675578 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675580 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675584 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675587 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675590 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675592 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675595 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675597 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675600 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675603 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675605 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675609 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675613 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 26 16:08:46.685840 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675615 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 16:08:46.686385 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675618 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 16:08:46.686385 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675621 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 16:08:46.686385 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675624 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 16:08:46.686385 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675627 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 16:08:46.686385 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.675631 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 16:08:46.686385 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.676244 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 26 16:08:46.686385 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.683359 2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 26 16:08:46.686385 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.683373 2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 26 16:08:46.686385 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.684123 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 26 16:08:46.686385 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.684130 2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 26 16:08:46.686385 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684187 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 16:08:46.686385 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684193 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 16:08:46.686385 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684196 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 16:08:46.686385 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684200 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684204 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684207 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684209 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684212 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684215 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684233 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684236 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684239 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684242 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684244 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684247 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684250 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684252 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684255 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684258 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684261 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684263 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684266 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684269 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 16:08:46.686729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684271 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684274 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684276 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684279 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684282 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684285 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684289 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684294 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684298 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684301 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684303 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684306 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684310 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684313 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684316 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684319 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684321 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684324 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684327 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 16:08:46.687236 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684329 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684332 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684334 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684337 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684340 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684342 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684345 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684347 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684350 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684353 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684355 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684359 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684362 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684365 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684368 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684371 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684374 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684376 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684379 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 16:08:46.687721 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684382 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684386 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684389 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684392 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684394 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684397 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684399 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684402 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684404 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684407 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684410 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684412 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684415 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684418 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684420 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684423 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684425 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684428 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684430 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684433 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684436 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 16:08:46.688187 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684438 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 16:08:46.688723 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684441 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 16:08:46.688723 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684444 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 16:08:46.688723 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684446 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 16:08:46.688723 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684449 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 16:08:46.688723 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684452 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 16:08:46.688723 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684454 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 16:08:46.688723 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684457 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 16:08:46.688723 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684459 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 16:08:46.688723 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684462 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 16:08:46.688723 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684464 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 16:08:46.688723 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684467 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 16:08:46.688723 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684469 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 16:08:46.688723 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684472 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 16:08:46.688723 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684475 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 16:08:46.688723 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684477 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 16:08:46.688723 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684479 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 16:08:46.689127 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.684484 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 26 16:08:46.689127 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684597 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 16:08:46.689127 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684601 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 16:08:46.689127 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684604 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 16:08:46.689127 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684607 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 16:08:46.689127 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684610 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 16:08:46.689127 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684613 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 16:08:46.689127 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684615 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 16:08:46.689127 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684618 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 16:08:46.689127 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684620 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 16:08:46.689127 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684623 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 16:08:46.689127 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684625 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 16:08:46.689127 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684628 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 16:08:46.689127 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684631 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 16:08:46.689127 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684634 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684636 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684639 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684641 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684645 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684649 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684651 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684654 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684658 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684661 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684664 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684667 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684670 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684673 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684675 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684678 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684682 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684685 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684688 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 16:08:46.689519 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684691 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684694 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684697 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684699 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684702 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684705 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684707 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684710 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684713 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684715 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684718 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684721 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684724 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684727 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684729 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684732 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684735 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684738 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684740 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 16:08:46.689979 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684744 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684746 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684749 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684752 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684755 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684758 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684760 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684763 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684765 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684768 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684771 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684774 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684776 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684779 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684781 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684783 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684786 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684788 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684791 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684793 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 16:08:46.690501 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684796 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684798 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684801 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684803 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684806 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684808 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684810 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684813 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684816 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684818 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684821 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684823 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684826 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684829 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684832 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684834 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684837 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684839 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684842 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684845 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 16:08:46.690983 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684847 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 16:08:46.691502 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684850 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 16:08:46.691502 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684852 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 16:08:46.691502 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684855 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 16:08:46.691502 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684858 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 16:08:46.691502 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684860 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 16:08:46.691502 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684863 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 16:08:46.691502 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:46.684865 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 16:08:46.691502 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.684870 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 26 16:08:46.691502 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.684978 2563 server.go:962] "Client rotation is on, will bootstrap in background" Jul 26 16:08:46.691502 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.687241 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 26 16:08:46.691502 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.688642 2563 server.go:1019] "Starting client certificate rotation" Jul 26 16:08:46.691502 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.688751 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 26 16:08:46.691502 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.688818 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 26 16:08:46.691827 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.691086 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 26 16:08:46.694566 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.694544 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 26 16:08:46.706627 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.706609 2563 log.go:25] "Validated CRI v1 runtime API" Jul 26 16:08:46.709522 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.709506 2563 log.go:25] "Validated CRI v1 image API" Jul 26 16:08:46.710360 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.710345 2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 26 16:08:46.713760 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.713739 2563 fs.go:135] Filesystem UUIDs: map[22fe117a-b7e9-40ca-a800-634df117c889:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 950dc090-9b49-4b8d-9a25-a17010d90a4d:/dev/nvme0n1p4] Jul 26 16:08:46.714290 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.713757 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 26 16:08:46.715933 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.715916 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 26 16:08:46.717968 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.717842 2563 manager.go:216] Machine: {Timestamp:2026-07-26 16:08:46.717375786 +0000 UTC m=+0.298462347 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3081943 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2971c0492a648ae445a9811a09e7ec SystemUUID:ec2971c0-492a-648a-e445-a9811a09e7ec BootID:8dc14d3d-d8bc-4141-99fa-1fc48878d4b2 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2b:1a:8d:8b:d5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2b:1a:8d:8b:d5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:1d:04:b4:cc:b4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 26 16:08:46.717968 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.717962 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 26 16:08:46.718087 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.718075 2563 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 26 16:08:46.720633 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.720608 2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 26 16:08:46.720760 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.720635 2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-70.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 26 16:08:46.721170 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.721161 2563 topology_manager.go:138] "Creating topology manager with none policy" Jul 26 16:08:46.721207 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.721173 2563 container_manager_linux.go:309] "Creating device plugin manager" Jul 26 16:08:46.721207 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.721180 2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 26 16:08:46.721784 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.721774 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 26 16:08:46.722852 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.722843 2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 26 16:08:46.725586 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.725576 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 26 16:08:46.725692 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.725682 2563 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 26 16:08:46.725730 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.725713 2563 kubelet.go:486] "Attempting to sync node with API server" Jul 26 16:08:46.725730 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.725723 2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 26 16:08:46.725819 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.725740 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 26 16:08:46.725819 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.725749 2563 kubelet.go:398] "Adding apiserver pod source" Jul 26 16:08:46.725819 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.725764 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 26 16:08:46.726177 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.726157 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 26 16:08:46.726258 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.726185 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 26 16:08:46.728200 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.728183 2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 26 16:08:46.730380 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.730360 2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 26 16:08:46.730453 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.730395 2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 26 16:08:46.732215 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.732198 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 26 16:08:46.732313 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.732216 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 26 16:08:46.732313 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.732249 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 26 16:08:46.732313 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.732255 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 26 16:08:46.732313 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.732265 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 26 16:08:46.732313 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.732271 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 26 16:08:46.732313 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.732279 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 26 16:08:46.732313 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.732284 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 26 16:08:46.732620 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.732323 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 26 16:08:46.732620 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.732332 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 26 16:08:46.732620 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.732354 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 26 16:08:46.732620 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.732362 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 26 16:08:46.733264 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.733250 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 26 16:08:46.733302 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.733267 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 26 16:08:46.736505 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.736490 2563 server.go:1268] "Started kubelet" Jul 26 16:08:46.736609 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.736573 2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 26 16:08:46.736665 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.736566 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 26 16:08:46.736712 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.736681 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 26 16:08:46.740342 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.740174 2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 26 16:08:46.740432 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:46.740375 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 26 16:08:46.740405 ip-10-0-130-70 systemd[1]: Started Kubernetes Kubelet. Jul 26 16:08:46.740644 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.740460 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-70.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 26 16:08:46.741188 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:46.740994 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-70.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 26 16:08:46.742533 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.742498 2563 server.go:310] "Adding debug handlers to kubelet server" Jul 26 16:08:46.745651 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:46.745630 2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 26 16:08:46.747668 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:46.746847 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-70.ec2.internal.18c5e2bc8b069852 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-70.ec2.internal,UID:ip-10-0-130-70.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-70.ec2.internal,},FirstTimestamp:2026-07-26 16:08:46.736463954 +0000 UTC m=+0.317550516,LastTimestamp:2026-07-26 16:08:46.736463954 +0000 UTC m=+0.317550516,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-70.ec2.internal,}" Jul 26 16:08:46.750717 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.750699 2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 26 16:08:46.750717 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.750709 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 26 16:08:46.750838 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:46.750776 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-70.ec2.internal\" not found" Jul 26 16:08:46.751526 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.751509 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 26 16:08:46.751526 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.751507 2563 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 26 16:08:46.751650 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.751536 2563 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 26 16:08:46.751650 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.751556 2563 factory.go:55] Registering systemd factory Jul 26 16:08:46.751650 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.751572 2563 factory.go:223] Registration of the systemd container factory successfully Jul 26 16:08:46.751837 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.751664 2563 reconstruct.go:97] "Volume reconstruction finished" Jul 26 16:08:46.751837 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.751675 2563 reconciler.go:29] "Reconciler: start to sync state" Jul 26 16:08:46.751837 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.751807 2563 factory.go:153] Registering CRI-O factory Jul 26 16:08:46.751837 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.751820 2563 factory.go:223] Registration of the crio container factory successfully Jul 26 16:08:46.752003 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.751874 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 26 16:08:46.752003 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.751904 2563 factory.go:103] Registering Raw factory Jul 26 16:08:46.752003 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.751918 2563 manager.go:1196] Started watching for new ooms in manager Jul 26 16:08:46.752333 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.752319 2563 manager.go:349] Starting recovery of all containers Jul 26 16:08:46.757063 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:46.757041 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 26 16:08:46.757272 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:46.757209 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-70.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 26 16:08:46.763094 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.763072 2563 manager.go:354] Recovery completed Jul 26 16:08:46.767903 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.767890 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 16:08:46.769653 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.769637 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasSufficientMemory" Jul 26 16:08:46.769736 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.769671 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 16:08:46.769736 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.769686 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasSufficientPID" Jul 26 16:08:46.770194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.770180 2563 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 26 16:08:46.770194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.770192 2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 26 16:08:46.770325 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.770192 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 16:08:46.770325 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.770207 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 26 16:08:46.771488 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.771466 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wp7js" Jul 26 16:08:46.771976 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.771959 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasSufficientMemory" Jul 26 16:08:46.772066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.771990 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 16:08:46.772066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.772005 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasSufficientPID" Jul 26 16:08:46.772308 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:46.772174 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-70.ec2.internal.18c5e2bc8d010845 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-70.ec2.internal,UID:ip-10-0-130-70.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-70.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-70.ec2.internal,},FirstTimestamp:2026-07-26 16:08:46.769653829 +0000 UTC m=+0.350740392,LastTimestamp:2026-07-26 16:08:46.769653829 +0000 UTC m=+0.350740392,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-70.ec2.internal,}" Jul 26 16:08:46.773294 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.773209 2563 policy_none.go:49] "None policy: Start" Jul 26 16:08:46.773294 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.773254 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 26 16:08:46.773294 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.773269 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 26 16:08:46.775121 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.775106 2563 policy_none.go:47] "Start" Jul 26 16:08:46.780094 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.780077 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wp7js" Jul 26 16:08:46.784421 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:46.784335 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-70.ec2.internal.18c5e2bc8d016482 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-70.ec2.internal,UID:ip-10-0-130-70.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-70.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-70.ec2.internal,},FirstTimestamp:2026-07-26 16:08:46.769677442 +0000 UTC m=+0.350764005,LastTimestamp:2026-07-26 16:08:46.769677442 +0000 UTC m=+0.350764005,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-70.ec2.internal,}" Jul 26 16:08:46.808761 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.808563 2563 manager.go:337] "Starting Device Plugin manager" Jul 26 16:08:46.808854 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:46.808787 2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 26 16:08:46.808854 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.808815 2563 server.go:87] "Starting device plugin registration server" Jul 26 16:08:46.809086 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.809074 2563 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 26 16:08:46.809123 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.809089 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 26 16:08:46.809192 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.809176 2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 26 16:08:46.809294 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.809283 2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 26 16:08:46.809356 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.809295 2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 26 16:08:46.809827 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:46.809811 2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 26 16:08:46.809910 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:46.809853 2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-70.ec2.internal\" not found" Jul 26 16:08:46.882853 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.882819 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 26 16:08:46.884818 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.884031 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 26 16:08:46.884818 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.884078 2563 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 26 16:08:46.884818 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.884105 2563 kubelet.go:2448] "Starting kubelet main sync loop" Jul 26 16:08:46.884818 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:46.884145 2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 26 16:08:46.886534 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.886512 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 16:08:46.909603 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.909582 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 16:08:46.910438 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.910422 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasSufficientMemory" Jul 26 16:08:46.910530 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.910455 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 16:08:46.910530 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.910469 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasSufficientPID" Jul 26 16:08:46.910530 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.910502 2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-70.ec2.internal" Jul 26 16:08:46.918257 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.918242 2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-70.ec2.internal" Jul 26 16:08:46.918305 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:46.918264 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-70.ec2.internal\": node \"ip-10-0-130-70.ec2.internal\" not found" Jul 26 16:08:46.933690 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:46.933614 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-70.ec2.internal\" not found" Jul 26 16:08:46.984963 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.984925 2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-70.ec2.internal"] Jul 26 16:08:46.985034 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.985006 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 16:08:46.986565 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.986540 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasSufficientMemory" Jul 26 16:08:46.986636 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.986572 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 16:08:46.986636 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.986582 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasSufficientPID" Jul 26 16:08:46.988886 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.988872 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 16:08:46.989021 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.989006 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal" Jul 26 16:08:46.989061 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.989042 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 16:08:46.989703 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.989686 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasSufficientMemory" Jul 26 16:08:46.989703 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.989694 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasSufficientMemory" Jul 26 16:08:46.989808 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.989711 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 16:08:46.989808 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.989720 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 16:08:46.989808 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.989737 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasSufficientPID" Jul 26 16:08:46.989808 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.989723 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasSufficientPID" Jul 26 16:08:46.991988 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.991967 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-70.ec2.internal" Jul 26 16:08:46.992094 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.991998 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 16:08:46.992728 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.992711 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasSufficientMemory" Jul 26 16:08:46.992803 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.992740 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 16:08:46.992803 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:46.992751 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasSufficientPID" Jul 26 16:08:47.018066 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:47.018043 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-70.ec2.internal\" not found" node="ip-10-0-130-70.ec2.internal" Jul 26 16:08:47.021790 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:47.021774 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-70.ec2.internal\" not found" node="ip-10-0-130-70.ec2.internal" Jul 26 16:08:47.034549 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:47.034528 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-70.ec2.internal\" not found" Jul 26 16:08:47.053364 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.053338 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f917223d7503166a38a92a8b80122252-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal\" (UID: \"f917223d7503166a38a92a8b80122252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal" Jul 26 16:08:47.053431 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.053371 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f917223d7503166a38a92a8b80122252-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal\" (UID: \"f917223d7503166a38a92a8b80122252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal" Jul 26 16:08:47.053431 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.053389 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e33f5784a8667eeea92ef51af7a40e20-config\") pod \"kube-apiserver-proxy-ip-10-0-130-70.ec2.internal\" (UID: \"e33f5784a8667eeea92ef51af7a40e20\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-70.ec2.internal" Jul 26 16:08:47.138158 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:47.138106 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-70.ec2.internal\" not found" Jul 26 16:08:47.153593 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.153564 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e33f5784a8667eeea92ef51af7a40e20-config\") pod \"kube-apiserver-proxy-ip-10-0-130-70.ec2.internal\" (UID: \"e33f5784a8667eeea92ef51af7a40e20\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-70.ec2.internal" Jul 26 16:08:47.153653 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.153597 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f917223d7503166a38a92a8b80122252-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal\" (UID: \"f917223d7503166a38a92a8b80122252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal" Jul 26 16:08:47.153653 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.153624 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f917223d7503166a38a92a8b80122252-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal\" (UID: \"f917223d7503166a38a92a8b80122252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal" Jul 26 16:08:47.153719 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.153678 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f917223d7503166a38a92a8b80122252-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal\" (UID: \"f917223d7503166a38a92a8b80122252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal" Jul 26 16:08:47.153719 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.153681 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e33f5784a8667eeea92ef51af7a40e20-config\") pod \"kube-apiserver-proxy-ip-10-0-130-70.ec2.internal\" (UID: \"e33f5784a8667eeea92ef51af7a40e20\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-70.ec2.internal" Jul 26 16:08:47.153787 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.153732 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f917223d7503166a38a92a8b80122252-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal\" (UID: \"f917223d7503166a38a92a8b80122252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal" Jul 26 16:08:47.238935 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:47.238851 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-70.ec2.internal\" not found" Jul 26 16:08:47.326070 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.326036 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-70.ec2.internal" Jul 26 16:08:47.326169 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.326038 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal" Jul 26 16:08:47.339550 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:47.339519 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-70.ec2.internal\" not found" Jul 26 16:08:47.444573 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:47.444523 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-70.ec2.internal\" not found" Jul 26 16:08:47.551096 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:47.551064 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-70.ec2.internal\" not found" Jul 26 16:08:47.588856 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.588831 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 16:08:47.653317 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:47.653285 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-70.ec2.internal\" not found" Jul 26 16:08:47.688739 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.688712 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 26 16:08:47.689194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.688871 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 26 16:08:47.689194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.688872 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 26 16:08:47.753628 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:47.753601 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-70.ec2.internal\" not found" Jul 26 16:08:47.771543 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.771516 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 16:08:47.786153 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.786106 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-25 16:03:47 +0000 UTC" deadline="2028-01-27 12:30:10.639352353 +0000 UTC" Jul 26 16:08:47.786153 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.786151 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13196h21m22.853205173s" Jul 26 16:08:47.839290 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.839263 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 16:08:47.840733 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.840715 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasSufficientMemory" Jul 26 16:08:47.840811 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.840746 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 16:08:47.840811 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.840757 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeHasSufficientPID" Jul 26 16:08:47.854745 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:47.854718 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-70.ec2.internal\" not found" Jul 26 16:08:47.955168 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:47.955142 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-70.ec2.internal\" not found" Jul 26 16:08:47.960428 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:47.960395 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf917223d7503166a38a92a8b80122252.slice/crio-bb09f20ddc77928825930519c49e5666fdbd6dcf24e0683ddc574582d4c3936b WatchSource:0}: Error finding container bb09f20ddc77928825930519c49e5666fdbd6dcf24e0683ddc574582d4c3936b: Status 404 returned error can't find the container with id bb09f20ddc77928825930519c49e5666fdbd6dcf24e0683ddc574582d4c3936b Jul 26 16:08:47.964799 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:47.964781 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 16:08:47.971132 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:47.971107 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33f5784a8667eeea92ef51af7a40e20.slice/crio-fa8a571f08be0536380c9347c6c2e0b7c42e1b588ff2a8879fc83842884066d3 WatchSource:0}: Error finding container fa8a571f08be0536380c9347c6c2e0b7c42e1b588ff2a8879fc83842884066d3: Status 404 returned error can't find the container with id fa8a571f08be0536380c9347c6c2e0b7c42e1b588ff2a8879fc83842884066d3 Jul 26 16:08:48.055978 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:48.055939 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-70.ec2.internal\" not found" Jul 26 16:08:48.156744 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:48.156665 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-70.ec2.internal\" not found" Jul 26 16:08:48.174868 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.174830 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 16:08:48.251106 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.251070 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal" Jul 26 16:08:48.264302 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.264281 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 26 16:08:48.265474 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.265460 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-70.ec2.internal" Jul 26 16:08:48.274478 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.274455 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 26 16:08:48.728449 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.728411 2563 apiserver.go:52] "Watching apiserver" Jul 26 16:08:48.738351 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.738316 2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 26 16:08:48.738865 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.738829 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-m4zvz","kube-system/konnectivity-agent-l7cw2","kube-system/kube-apiserver-proxy-ip-10-0-130-70.ec2.internal","openshift-dns/node-resolver-bqnx2","openshift-image-registry/node-ca-2nwxs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal","openshift-multus/multus-knl4q","openshift-network-diagnostics/network-check-target-dndsz","kube-system/global-pull-secret-syncer-hs2qb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl","openshift-cluster-node-tuning-operator/tuned-mt7v8","openshift-multus/multus-additional-cni-plugins-bd5xf","openshift-multus/network-metrics-daemon-qd8ng","openshift-network-operator/iptables-alerter-m98wg"] Jul 26 16:08:48.740659 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.740631 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 16:08:48.742479 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.742453 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.744623 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.744595 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:08:48.744727 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:48.744672 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8ng" podUID="bc7a4aef-1953-45ab-84db-8338395b0b25" Jul 26 16:08:48.746098 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.745902 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 26 16:08:48.746098 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.745921 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 26 16:08:48.746098 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.745966 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 26 16:08:48.746421 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.746115 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 26 16:08:48.746421 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.746280 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w2j8x\"" Jul 26 16:08:48.747474 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.747452 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m98wg" Jul 26 16:08:48.749983 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.749952 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bqnx2" Jul 26 16:08:48.750362 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.750337 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 26 16:08:48.750477 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.750418 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 26 16:08:48.750672 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.750652 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jdg84\"" Jul 26 16:08:48.750672 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.750669 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 26 16:08:48.751538 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.751521 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 26 16:08:48.753120 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.752388 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2nwxs" Jul 26 16:08:48.753120 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.752532 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 26 16:08:48.753120 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.752902 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 26 16:08:48.753120 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.752990 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9gdj6\"" Jul 26 16:08:48.755742 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.754993 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:08:48.755742 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:48.755074 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dndsz" podUID="1d66c1fe-74b0-44a3-b892-bd68dd3ffc51" Jul 26 16:08:48.755742 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.755204 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 26 16:08:48.755742 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.755437 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 26 16:08:48.755742 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.755557 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 26 16:08:48.755742 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.755645 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dkfmz\"" Jul 26 16:08:48.757308 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.757273 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:08:48.757407 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:48.757341 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hs2qb" podUID="9f8db869-87b0-4a2c-b292-b78f22f3a391" Jul 26 16:08:48.759972 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.759953 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.762873 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.762624 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l7cw2" Jul 26 16:08:48.763273 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763251 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f-hosts-file\") pod \"node-resolver-bqnx2\" (UID: \"8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f\") " pod="openshift-dns/node-resolver-bqnx2" Jul 26 16:08:48.763394 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763308 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-host-var-lib-kubelet\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.763394 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763340 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-host-run-multus-certs\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.763394 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763366 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs\") pod \"network-metrics-daemon-qd8ng\" (UID: \"bc7a4aef-1953-45ab-84db-8338395b0b25\") " pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:08:48.763565 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763393 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd6xc\" (UniqueName: \"kubernetes.io/projected/bc7a4aef-1953-45ab-84db-8338395b0b25-kube-api-access-zd6xc\") pod \"network-metrics-daemon-qd8ng\" (UID: \"bc7a4aef-1953-45ab-84db-8338395b0b25\") " pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:08:48.763565 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763426 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8flln\" (UniqueName: \"kubernetes.io/projected/8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f-kube-api-access-8flln\") pod \"node-resolver-bqnx2\" (UID: \"8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f\") " pod="openshift-dns/node-resolver-bqnx2" Jul 26 16:08:48.763565 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763449 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8fd86d4-e511-4f8e-962d-8c6726ba4cd1-serviceca\") pod \"node-ca-2nwxs\" (UID: \"e8fd86d4-e511-4f8e-962d-8c6726ba4cd1\") " pod="openshift-image-registry/node-ca-2nwxs" Jul 26 16:08:48.763565 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763474 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-host-var-lib-cni-bin\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.763565 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763501 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-multus-cni-dir\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.763565 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763526 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7q97\" (UniqueName: \"kubernetes.io/projected/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-kube-api-access-j7q97\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.763565 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763550 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-multus-daemon-config\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.763845 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763575 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32afde23-1522-43c3-93dc-fd4a60a27d5f-host-slash\") pod \"iptables-alerter-m98wg\" (UID: \"32afde23-1522-43c3-93dc-fd4a60a27d5f\") " pod="openshift-network-operator/iptables-alerter-m98wg" Jul 26 16:08:48.763845 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763599 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f-tmp-dir\") pod \"node-resolver-bqnx2\" (UID: \"8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f\") " pod="openshift-dns/node-resolver-bqnx2" Jul 26 16:08:48.763845 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763625 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-system-cni-dir\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.763845 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763648 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-cnibin\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.763845 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763672 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-multus-socket-dir-parent\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.763845 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763706 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-host-run-k8s-cni-cncf-io\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.763845 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763744 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-host-var-lib-cni-multus\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.763845 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763779 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-multus-conf-dir\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.763845 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763810 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32afde23-1522-43c3-93dc-fd4a60a27d5f-iptables-alerter-script\") pod \"iptables-alerter-m98wg\" (UID: \"32afde23-1522-43c3-93dc-fd4a60a27d5f\") " pod="openshift-network-operator/iptables-alerter-m98wg" Jul 26 16:08:48.764228 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763836 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8lph\" (UniqueName: \"kubernetes.io/projected/32afde23-1522-43c3-93dc-fd4a60a27d5f-kube-api-access-l8lph\") pod \"iptables-alerter-m98wg\" (UID: \"32afde23-1522-43c3-93dc-fd4a60a27d5f\") " pod="openshift-network-operator/iptables-alerter-m98wg" Jul 26 16:08:48.764228 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763870 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-cni-binary-copy\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.764228 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763896 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-etc-kubernetes\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.764228 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763919 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8fd86d4-e511-4f8e-962d-8c6726ba4cd1-host\") pod \"node-ca-2nwxs\" (UID: \"e8fd86d4-e511-4f8e-962d-8c6726ba4cd1\") " pod="openshift-image-registry/node-ca-2nwxs" Jul 26 16:08:48.764228 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763944 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw79d\" (UniqueName: \"kubernetes.io/projected/e8fd86d4-e511-4f8e-962d-8c6726ba4cd1-kube-api-access-jw79d\") pod \"node-ca-2nwxs\" (UID: \"e8fd86d4-e511-4f8e-962d-8c6726ba4cd1\") " pod="openshift-image-registry/node-ca-2nwxs" Jul 26 16:08:48.764228 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763971 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-os-release\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.764228 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.763995 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-host-run-netns\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.764228 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.764018 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-hostroot\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.764519 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.764343 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hq5gc\"" Jul 26 16:08:48.764519 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.764413 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 26 16:08:48.764587 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.764557 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 26 16:08:48.764651 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.764633 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 26 16:08:48.764804 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.764786 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 26 16:08:48.764919 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.764885 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 26 16:08:48.765057 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.765037 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 26 16:08:48.766263 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.766167 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.766457 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.766378 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 26 16:08:48.767414 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.766848 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 26 16:08:48.767502 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.767488 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bcs97\"" Jul 26 16:08:48.768670 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.768648 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 26 16:08:48.770950 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.769486 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 26 16:08:48.770950 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.769552 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 26 16:08:48.771352 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.771327 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vltp9\"" Jul 26 16:08:48.771449 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.771428 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 26 16:08:48.772344 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.772319 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.775919 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.775892 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 26 16:08:48.776014 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.775894 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 26 16:08:48.776014 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.775909 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.776123 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.776016 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ptq2r\"" Jul 26 16:08:48.778693 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.778673 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qv8fl\"" Jul 26 16:08:48.778898 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.778882 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 26 16:08:48.779141 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.779117 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 26 16:08:48.787570 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.787541 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4jh9m" Jul 26 16:08:48.795947 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.795919 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4jh9m" Jul 26 16:08:48.852862 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.852825 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 26 16:08:48.865053 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865028 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8fd86d4-e511-4f8e-962d-8c6726ba4cd1-host\") pod \"node-ca-2nwxs\" (UID: \"e8fd86d4-e511-4f8e-962d-8c6726ba4cd1\") " pod="openshift-image-registry/node-ca-2nwxs" Jul 26 16:08:48.865053 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865060 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-host-run-netns\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.865284 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865087 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/456b66bc-bd4c-45da-96d6-6d1debc7987e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.865284 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865112 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/456b66bc-bd4c-45da-96d6-6d1debc7987e-registration-dir\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.865284 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865138 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/456b66bc-bd4c-45da-96d6-6d1debc7987e-device-dir\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.865284 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865164 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-systemd\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.865284 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865195 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-run-ovn\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.865284 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865214 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-host-run-netns\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.865284 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865225 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-host-run-multus-certs\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.865638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865294 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-host-run-multus-certs\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.865638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865291 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-cni-bin\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.865638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865162 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8fd86d4-e511-4f8e-962d-8c6726ba4cd1-host\") pod \"node-ca-2nwxs\" (UID: \"e8fd86d4-e511-4f8e-962d-8c6726ba4cd1\") " pod="openshift-image-registry/node-ca-2nwxs" Jul 26 16:08:48.865638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865336 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-env-overrides\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.865638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865369 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd6xc\" (UniqueName: \"kubernetes.io/projected/bc7a4aef-1953-45ab-84db-8338395b0b25-kube-api-access-zd6xc\") pod \"network-metrics-daemon-qd8ng\" (UID: \"bc7a4aef-1953-45ab-84db-8338395b0b25\") " pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:08:48.865638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865395 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8flln\" (UniqueName: \"kubernetes.io/projected/8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f-kube-api-access-8flln\") pod \"node-resolver-bqnx2\" (UID: \"8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f\") " pod="openshift-dns/node-resolver-bqnx2" Jul 26 16:08:48.865638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865420 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8fd86d4-e511-4f8e-962d-8c6726ba4cd1-serviceca\") pod \"node-ca-2nwxs\" (UID: \"e8fd86d4-e511-4f8e-962d-8c6726ba4cd1\") " pod="openshift-image-registry/node-ca-2nwxs" Jul 26 16:08:48.865638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865446 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7f65fa5-91ab-49cf-ab27-f4af947b3bbe-konnectivity-ca\") pod \"konnectivity-agent-l7cw2\" (UID: \"f7f65fa5-91ab-49cf-ab27-f4af947b3bbe\") " pod="kube-system/konnectivity-agent-l7cw2" Jul 26 16:08:48.865638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865473 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-kubernetes\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.865638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865505 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-run\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.865638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865531 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d83251cf-1af1-438a-9525-a79558f5f48f-system-cni-dir\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.865638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865557 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7f65fa5-91ab-49cf-ab27-f4af947b3bbe-agent-certs\") pod \"konnectivity-agent-l7cw2\" (UID: \"f7f65fa5-91ab-49cf-ab27-f4af947b3bbe\") " pod="kube-system/konnectivity-agent-l7cw2" Jul 26 16:08:48.865638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865583 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/456b66bc-bd4c-45da-96d6-6d1debc7987e-etc-selinux\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.865638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865618 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-sysctl-conf\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.865638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865642 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-sys\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.866425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865666 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-lib-modules\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.866425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865711 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d83251cf-1af1-438a-9525-a79558f5f48f-cnibin\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.866425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865738 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d83251cf-1af1-438a-9525-a79558f5f48f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.866425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865764 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7q97\" (UniqueName: \"kubernetes.io/projected/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-kube-api-access-j7q97\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.866425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865787 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/456b66bc-bd4c-45da-96d6-6d1debc7987e-socket-dir\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.866425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865811 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-etc-openvswitch\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.866425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865845 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-node-log\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.866425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865873 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-multus-daemon-config\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.866425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865898 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-system-cni-dir\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.866425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865925 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-cnibin\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.866425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865962 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-host-run-k8s-cni-cncf-io\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.866425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.865988 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-host-var-lib-cni-multus\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.866425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866012 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/456b66bc-bd4c-45da-96d6-6d1debc7987e-sys-fs\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.866425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866036 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d83251cf-1af1-438a-9525-a79558f5f48f-cni-binary-copy\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.866425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866061 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-var-lib-openvswitch\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.866425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866085 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-kubelet\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.866425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866111 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-slash\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.867269 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866136 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-cni-netd\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.867269 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866162 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.867269 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866188 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-ovn-node-metrics-cert\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.867269 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866225 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32afde23-1522-43c3-93dc-fd4a60a27d5f-iptables-alerter-script\") pod \"iptables-alerter-m98wg\" (UID: \"32afde23-1522-43c3-93dc-fd4a60a27d5f\") " pod="openshift-network-operator/iptables-alerter-m98wg" Jul 26 16:08:48.867269 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866252 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8lph\" (UniqueName: \"kubernetes.io/projected/32afde23-1522-43c3-93dc-fd4a60a27d5f-kube-api-access-l8lph\") pod \"iptables-alerter-m98wg\" (UID: \"32afde23-1522-43c3-93dc-fd4a60a27d5f\") " pod="openshift-network-operator/iptables-alerter-m98wg" Jul 26 16:08:48.867269 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866278 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-cni-binary-copy\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.867269 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866322 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-etc-kubernetes\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.867269 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866350 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-sysconfig\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.867269 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866377 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw79d\" (UniqueName: \"kubernetes.io/projected/e8fd86d4-e511-4f8e-962d-8c6726ba4cd1-kube-api-access-jw79d\") pod \"node-ca-2nwxs\" (UID: \"e8fd86d4-e511-4f8e-962d-8c6726ba4cd1\") " pod="openshift-image-registry/node-ca-2nwxs" Jul 26 16:08:48.867269 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866407 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-os-release\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.867269 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866431 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-hostroot\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.867269 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866441 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8fd86d4-e511-4f8e-962d-8c6726ba4cd1-serviceca\") pod \"node-ca-2nwxs\" (UID: \"e8fd86d4-e511-4f8e-962d-8c6726ba4cd1\") " pod="openshift-image-registry/node-ca-2nwxs" Jul 26 16:08:48.867269 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866458 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9f8db869-87b0-4a2c-b292-b78f22f3a391-kubelet-config\") pod \"global-pull-secret-syncer-hs2qb\" (UID: \"9f8db869-87b0-4a2c-b292-b78f22f3a391\") " pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:08:48.867269 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866488 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9f8db869-87b0-4a2c-b292-b78f22f3a391-dbus\") pod \"global-pull-secret-syncer-hs2qb\" (UID: \"9f8db869-87b0-4a2c-b292-b78f22f3a391\") " pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:08:48.867269 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866529 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret\") pod \"global-pull-secret-syncer-hs2qb\" (UID: \"9f8db869-87b0-4a2c-b292-b78f22f3a391\") " pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:08:48.867269 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866593 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-ovnkube-config\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.867269 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866624 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f-hosts-file\") pod \"node-resolver-bqnx2\" (UID: \"8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f\") " pod="openshift-dns/node-resolver-bqnx2" Jul 26 16:08:48.868066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866660 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-host-var-lib-kubelet\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.868066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866739 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-cnibin\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.868066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866748 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-modprobe-d\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.868066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866798 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-host-run-k8s-cni-cncf-io\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.868066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866807 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-etc-kubernetes\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.868066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866846 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f-hosts-file\") pod \"node-resolver-bqnx2\" (UID: \"8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f\") " pod="openshift-dns/node-resolver-bqnx2" Jul 26 16:08:48.868066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.866850 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-os-release\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.868066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867104 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-host-var-lib-cni-multus\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.868066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867331 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-systemd-units\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.868066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867368 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs\") pod \"network-metrics-daemon-qd8ng\" (UID: \"bc7a4aef-1953-45ab-84db-8338395b0b25\") " pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:08:48.868066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867385 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-host-var-lib-cni-bin\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.868066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867395 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32afde23-1522-43c3-93dc-fd4a60a27d5f-iptables-alerter-script\") pod \"iptables-alerter-m98wg\" (UID: \"32afde23-1522-43c3-93dc-fd4a60a27d5f\") " pod="openshift-network-operator/iptables-alerter-m98wg" Jul 26 16:08:48.868066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867394 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-cni-binary-copy\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.868066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867415 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlmns\" (UniqueName: \"kubernetes.io/projected/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-kube-api-access-wlmns\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.868066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867441 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-host-var-lib-cni-bin\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.868066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867446 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-multus-daemon-config\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.868066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867454 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/665e14a9-02c7-45b1-87d2-d9e98e559d12-tmp\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.868066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867445 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-hostroot\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.869030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867517 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-system-cni-dir\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.869030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867516 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-host-var-lib-kubelet\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.869030 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:48.867562 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 16:08:48.869030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867574 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-run-netns\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.869030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867615 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-run-openvswitch\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.869030 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:48.867644 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs podName:bc7a4aef-1953-45ab-84db-8338395b0b25 nodeName:}" failed. No retries permitted until 2026-07-26 16:08:49.367624065 +0000 UTC m=+2.948710615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs") pod "network-metrics-daemon-qd8ng" (UID: "bc7a4aef-1953-45ab-84db-8338395b0b25") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 16:08:48.869030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867725 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-log-socket\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.869030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867762 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.869030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867819 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-multus-cni-dir\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.869030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867879 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjgxp\" (UniqueName: \"kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp\") pod \"network-check-target-dndsz\" (UID: \"1d66c1fe-74b0-44a3-b892-bd68dd3ffc51\") " pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:08:48.869030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867907 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-run-systemd\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.869030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867952 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32afde23-1522-43c3-93dc-fd4a60a27d5f-host-slash\") pod \"iptables-alerter-m98wg\" (UID: \"32afde23-1522-43c3-93dc-fd4a60a27d5f\") " pod="openshift-network-operator/iptables-alerter-m98wg" Jul 26 16:08:48.869030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.867991 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f-tmp-dir\") pod \"node-resolver-bqnx2\" (UID: \"8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f\") " pod="openshift-dns/node-resolver-bqnx2" Jul 26 16:08:48.869030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.868019 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-multus-socket-dir-parent\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.869030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.868062 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-sysctl-d\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.869030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.868089 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-tuned\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.869030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.868115 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d83251cf-1af1-438a-9525-a79558f5f48f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.869925 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.868118 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-multus-cni-dir\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.869925 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.868260 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgpnf\" (UniqueName: \"kubernetes.io/projected/d83251cf-1af1-438a-9525-a79558f5f48f-kube-api-access-jgpnf\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.869925 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.868305 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-multus-conf-dir\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.869925 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.868331 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-var-lib-kubelet\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.869925 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.868362 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-host\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.869925 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.868386 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t42t\" (UniqueName: \"kubernetes.io/projected/665e14a9-02c7-45b1-87d2-d9e98e559d12-kube-api-access-8t42t\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.869925 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.868414 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d83251cf-1af1-438a-9525-a79558f5f48f-os-release\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.869925 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.868441 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t47nt\" (UniqueName: \"kubernetes.io/projected/456b66bc-bd4c-45da-96d6-6d1debc7987e-kube-api-access-t47nt\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.869925 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.868453 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32afde23-1522-43c3-93dc-fd4a60a27d5f-host-slash\") pod \"iptables-alerter-m98wg\" (UID: \"32afde23-1522-43c3-93dc-fd4a60a27d5f\") " pod="openshift-network-operator/iptables-alerter-m98wg" Jul 26 16:08:48.869925 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.868466 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d83251cf-1af1-438a-9525-a79558f5f48f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.869925 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.868495 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-ovnkube-script-lib\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.869925 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.868392 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f-tmp-dir\") pod \"node-resolver-bqnx2\" (UID: \"8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f\") " pod="openshift-dns/node-resolver-bqnx2" Jul 26 16:08:48.869925 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.868498 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-multus-socket-dir-parent\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.869925 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.868598 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-multus-conf-dir\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.884851 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.884472 2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 26 16:08:48.888713 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.888615 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7q97\" (UniqueName: \"kubernetes.io/projected/d9ec0a41-3fb0-4b8e-a957-5a300790d89b-kube-api-access-j7q97\") pod \"multus-knl4q\" (UID: \"d9ec0a41-3fb0-4b8e-a957-5a300790d89b\") " pod="openshift-multus/multus-knl4q" Jul 26 16:08:48.888713 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.888633 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd6xc\" (UniqueName: \"kubernetes.io/projected/bc7a4aef-1953-45ab-84db-8338395b0b25-kube-api-access-zd6xc\") pod \"network-metrics-daemon-qd8ng\" (UID: \"bc7a4aef-1953-45ab-84db-8338395b0b25\") " pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:08:48.888713 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.888680 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8lph\" (UniqueName: \"kubernetes.io/projected/32afde23-1522-43c3-93dc-fd4a60a27d5f-kube-api-access-l8lph\") pod \"iptables-alerter-m98wg\" (UID: \"32afde23-1522-43c3-93dc-fd4a60a27d5f\") " pod="openshift-network-operator/iptables-alerter-m98wg" Jul 26 16:08:48.888952 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.888758 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw79d\" (UniqueName: \"kubernetes.io/projected/e8fd86d4-e511-4f8e-962d-8c6726ba4cd1-kube-api-access-jw79d\") pod \"node-ca-2nwxs\" (UID: \"e8fd86d4-e511-4f8e-962d-8c6726ba4cd1\") " pod="openshift-image-registry/node-ca-2nwxs" Jul 26 16:08:48.889685 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.889430 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8flln\" (UniqueName: \"kubernetes.io/projected/8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f-kube-api-access-8flln\") pod \"node-resolver-bqnx2\" (UID: \"8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f\") " pod="openshift-dns/node-resolver-bqnx2" Jul 26 16:08:48.890912 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.890769 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-70.ec2.internal" event={"ID":"e33f5784a8667eeea92ef51af7a40e20","Type":"ContainerStarted","Data":"fa8a571f08be0536380c9347c6c2e0b7c42e1b588ff2a8879fc83842884066d3"} Jul 26 16:08:48.893324 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.893142 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal" event={"ID":"f917223d7503166a38a92a8b80122252","Type":"ContainerStarted","Data":"bb09f20ddc77928825930519c49e5666fdbd6dcf24e0683ddc574582d4c3936b"} Jul 26 16:08:48.969337 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969301 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-sys\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.969511 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969348 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-lib-modules\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.969511 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969378 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d83251cf-1af1-438a-9525-a79558f5f48f-cnibin\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.969511 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969406 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d83251cf-1af1-438a-9525-a79558f5f48f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.969511 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969435 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/456b66bc-bd4c-45da-96d6-6d1debc7987e-socket-dir\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.969511 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969443 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-sys\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.969511 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969462 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-etc-openvswitch\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.969511 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969471 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d83251cf-1af1-438a-9525-a79558f5f48f-cnibin\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.969511 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969500 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-lib-modules\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.969511 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969506 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-etc-openvswitch\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.969979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969506 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-node-log\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.969979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969568 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-node-log\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.969979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969581 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/456b66bc-bd4c-45da-96d6-6d1debc7987e-sys-fs\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.969979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969618 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/456b66bc-bd4c-45da-96d6-6d1debc7987e-socket-dir\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.969979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969636 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/456b66bc-bd4c-45da-96d6-6d1debc7987e-sys-fs\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.969979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969646 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d83251cf-1af1-438a-9525-a79558f5f48f-cni-binary-copy\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.969979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969675 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-var-lib-openvswitch\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.969979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969700 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-kubelet\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.969979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969735 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-slash\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.969979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969749 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-var-lib-openvswitch\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.969979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969777 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-cni-netd\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.969979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969795 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-kubelet\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.969979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969807 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.969979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969826 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-ovn-node-metrics-cert\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.969979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969840 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-slash\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.969979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969852 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-sysconfig\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.969979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969879 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9f8db869-87b0-4a2c-b292-b78f22f3a391-kubelet-config\") pod \"global-pull-secret-syncer-hs2qb\" (UID: \"9f8db869-87b0-4a2c-b292-b78f22f3a391\") " pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:08:48.970849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969886 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.970849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969903 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9f8db869-87b0-4a2c-b292-b78f22f3a391-dbus\") pod \"global-pull-secret-syncer-hs2qb\" (UID: \"9f8db869-87b0-4a2c-b292-b78f22f3a391\") " pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:08:48.970849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969929 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-cni-netd\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.970849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969933 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret\") pod \"global-pull-secret-syncer-hs2qb\" (UID: \"9f8db869-87b0-4a2c-b292-b78f22f3a391\") " pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:08:48.970849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969958 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-ovnkube-config\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.970849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969981 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-modprobe-d\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.970849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969999 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-systemd-units\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.970849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970025 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlmns\" (UniqueName: \"kubernetes.io/projected/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-kube-api-access-wlmns\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.970849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970042 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/665e14a9-02c7-45b1-87d2-d9e98e559d12-tmp\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.970849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970060 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-run-netns\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.970849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970076 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-run-openvswitch\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.970849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970090 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-log-socket\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.970849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970105 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.970849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970130 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjgxp\" (UniqueName: \"kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp\") pod \"network-check-target-dndsz\" (UID: \"1d66c1fe-74b0-44a3-b892-bd68dd3ffc51\") " pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:08:48.970849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970142 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d83251cf-1af1-438a-9525-a79558f5f48f-cni-binary-copy\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.970849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970167 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-run-systemd\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.970849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970186 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d83251cf-1af1-438a-9525-a79558f5f48f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.971780 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970209 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-sysctl-d\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.971780 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970218 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-run-systemd\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.971780 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970259 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-systemd-units\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.971780 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:48.970315 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 16:08:48.971780 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970337 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-sysctl-d\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.971780 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:48.970387 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret podName:9f8db869-87b0-4a2c-b292-b78f22f3a391 nodeName:}" failed. No retries permitted until 2026-07-26 16:08:49.470367702 +0000 UTC m=+3.051454252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret") pod "global-pull-secret-syncer-hs2qb" (UID: "9f8db869-87b0-4a2c-b292-b78f22f3a391") : object "kube-system"/"original-pull-secret" not registered Jul 26 16:08:48.971780 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970408 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-modprobe-d\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.971780 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.969981 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-sysconfig\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.971780 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970435 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9f8db869-87b0-4a2c-b292-b78f22f3a391-kubelet-config\") pod \"global-pull-secret-syncer-hs2qb\" (UID: \"9f8db869-87b0-4a2c-b292-b78f22f3a391\") " pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:08:48.971780 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970483 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9f8db869-87b0-4a2c-b292-b78f22f3a391-dbus\") pod \"global-pull-secret-syncer-hs2qb\" (UID: \"9f8db869-87b0-4a2c-b292-b78f22f3a391\") " pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:08:48.971780 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970489 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-log-socket\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.971780 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970526 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-run-netns\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.971780 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970531 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-run-openvswitch\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.971780 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970563 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-tuned\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.971780 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970591 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d83251cf-1af1-438a-9525-a79558f5f48f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.971780 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970620 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgpnf\" (UniqueName: \"kubernetes.io/projected/d83251cf-1af1-438a-9525-a79558f5f48f-kube-api-access-jgpnf\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.971780 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970646 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-var-lib-kubelet\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.972621 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970671 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-host\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.972621 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970695 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t42t\" (UniqueName: \"kubernetes.io/projected/665e14a9-02c7-45b1-87d2-d9e98e559d12-kube-api-access-8t42t\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.972621 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970719 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d83251cf-1af1-438a-9525-a79558f5f48f-os-release\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.972621 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970745 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t47nt\" (UniqueName: \"kubernetes.io/projected/456b66bc-bd4c-45da-96d6-6d1debc7987e-kube-api-access-t47nt\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.972621 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970745 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.972621 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970773 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d83251cf-1af1-438a-9525-a79558f5f48f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.972621 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970799 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-ovnkube-script-lib\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.972621 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970811 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-host\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.972621 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970828 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/456b66bc-bd4c-45da-96d6-6d1debc7987e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.972621 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970854 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/456b66bc-bd4c-45da-96d6-6d1debc7987e-registration-dir\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.972621 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970857 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-ovnkube-config\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.972621 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970880 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/456b66bc-bd4c-45da-96d6-6d1debc7987e-device-dir\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.972621 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970905 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-systemd\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.972621 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970930 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-run-ovn\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.972621 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970958 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-cni-bin\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.972621 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.970983 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-env-overrides\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.972621 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971022 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7f65fa5-91ab-49cf-ab27-f4af947b3bbe-konnectivity-ca\") pod \"konnectivity-agent-l7cw2\" (UID: \"f7f65fa5-91ab-49cf-ab27-f4af947b3bbe\") " pod="kube-system/konnectivity-agent-l7cw2" Jul 26 16:08:48.973436 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971048 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-kubernetes\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.973436 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971073 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-run\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.973436 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971101 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d83251cf-1af1-438a-9525-a79558f5f48f-system-cni-dir\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.973436 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971129 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7f65fa5-91ab-49cf-ab27-f4af947b3bbe-agent-certs\") pod \"konnectivity-agent-l7cw2\" (UID: \"f7f65fa5-91ab-49cf-ab27-f4af947b3bbe\") " pod="kube-system/konnectivity-agent-l7cw2" Jul 26 16:08:48.973436 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971153 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/456b66bc-bd4c-45da-96d6-6d1debc7987e-etc-selinux\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.973436 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971150 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-var-lib-kubelet\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.973436 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971179 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-sysctl-conf\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.973436 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971215 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-systemd\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.973436 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971260 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-run-ovn\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.973436 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971268 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d83251cf-1af1-438a-9525-a79558f5f48f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.973436 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971318 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-host-cni-bin\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.973436 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971364 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-sysctl-conf\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.973436 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971369 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d83251cf-1af1-438a-9525-a79558f5f48f-os-release\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.973436 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971588 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-run\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.973436 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971689 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d83251cf-1af1-438a-9525-a79558f5f48f-system-cni-dir\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.973436 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971725 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-env-overrides\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.973436 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971747 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-kubernetes\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.974219 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971794 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/456b66bc-bd4c-45da-96d6-6d1debc7987e-registration-dir\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.974219 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971837 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/456b66bc-bd4c-45da-96d6-6d1debc7987e-etc-selinux\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.974219 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971844 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/456b66bc-bd4c-45da-96d6-6d1debc7987e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.974219 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.971892 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/456b66bc-bd4c-45da-96d6-6d1debc7987e-device-dir\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:48.974219 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.972016 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-ovnkube-script-lib\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.974219 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.972175 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7f65fa5-91ab-49cf-ab27-f4af947b3bbe-konnectivity-ca\") pod \"konnectivity-agent-l7cw2\" (UID: \"f7f65fa5-91ab-49cf-ab27-f4af947b3bbe\") " pod="kube-system/konnectivity-agent-l7cw2" Jul 26 16:08:48.974219 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.972440 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d83251cf-1af1-438a-9525-a79558f5f48f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.974219 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.973809 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/665e14a9-02c7-45b1-87d2-d9e98e559d12-etc-tuned\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.974219 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.973947 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/665e14a9-02c7-45b1-87d2-d9e98e559d12-tmp\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.974677 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.974218 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-ovn-node-metrics-cert\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.974677 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.974514 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7f65fa5-91ab-49cf-ab27-f4af947b3bbe-agent-certs\") pod \"konnectivity-agent-l7cw2\" (UID: \"f7f65fa5-91ab-49cf-ab27-f4af947b3bbe\") " pod="kube-system/konnectivity-agent-l7cw2" Jul 26 16:08:48.983302 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:48.983245 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 16:08:48.983302 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:48.983274 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 16:08:48.983302 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:48.983289 2563 projected.go:196] Error preparing data for projected volume kube-api-access-fjgxp for pod openshift-network-diagnostics/network-check-target-dndsz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 16:08:48.983520 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:48.983390 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp podName:1d66c1fe-74b0-44a3-b892-bd68dd3ffc51 nodeName:}" failed. No retries permitted until 2026-07-26 16:08:49.483369402 +0000 UTC m=+3.064455972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fjgxp" (UniqueName: "kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp") pod "network-check-target-dndsz" (UID: "1d66c1fe-74b0-44a3-b892-bd68dd3ffc51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 16:08:48.985783 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.985761 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlmns\" (UniqueName: \"kubernetes.io/projected/7ba16aca-ddb5-4c58-a2a5-89d8b52150a3-kube-api-access-wlmns\") pod \"ovnkube-node-m4zvz\" (UID: \"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:48.986006 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.985974 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t42t\" (UniqueName: \"kubernetes.io/projected/665e14a9-02c7-45b1-87d2-d9e98e559d12-kube-api-access-8t42t\") pod \"tuned-mt7v8\" (UID: \"665e14a9-02c7-45b1-87d2-d9e98e559d12\") " pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:48.986430 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.986406 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgpnf\" (UniqueName: \"kubernetes.io/projected/d83251cf-1af1-438a-9525-a79558f5f48f-kube-api-access-jgpnf\") pod \"multus-additional-cni-plugins-bd5xf\" (UID: \"d83251cf-1af1-438a-9525-a79558f5f48f\") " pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:48.988221 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:48.988200 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t47nt\" (UniqueName: \"kubernetes.io/projected/456b66bc-bd4c-45da-96d6-6d1debc7987e-kube-api-access-t47nt\") pod \"aws-ebs-csi-driver-node-4jjvl\" (UID: \"456b66bc-bd4c-45da-96d6-6d1debc7987e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:49.057991 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:49.057953 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-knl4q" Jul 26 16:08:49.068663 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:49.068631 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m98wg" Jul 26 16:08:49.080076 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:49.080046 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bqnx2" Jul 26 16:08:49.086932 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:49.086902 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2nwxs" Jul 26 16:08:49.095053 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:49.095031 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:08:49.112003 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:49.111972 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l7cw2" Jul 26 16:08:49.121023 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:49.120992 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" Jul 26 16:08:49.129059 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:49.129039 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" Jul 26 16:08:49.135167 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:49.135148 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bd5xf" Jul 26 16:08:49.374886 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:49.374849 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs\") pod \"network-metrics-daemon-qd8ng\" (UID: \"bc7a4aef-1953-45ab-84db-8338395b0b25\") " pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:08:49.375082 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:49.375051 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 16:08:49.375154 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:49.375141 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs podName:bc7a4aef-1953-45ab-84db-8338395b0b25 nodeName:}" failed. No retries permitted until 2026-07-26 16:08:50.375118598 +0000 UTC m=+3.956205169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs") pod "network-metrics-daemon-qd8ng" (UID: "bc7a4aef-1953-45ab-84db-8338395b0b25") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 16:08:49.476108 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:49.476070 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret\") pod \"global-pull-secret-syncer-hs2qb\" (UID: \"9f8db869-87b0-4a2c-b292-b78f22f3a391\") " pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:08:49.476328 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:49.476265 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 16:08:49.476399 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:49.476361 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret podName:9f8db869-87b0-4a2c-b292-b78f22f3a391 nodeName:}" failed. No retries permitted until 2026-07-26 16:08:50.476332344 +0000 UTC m=+4.057418912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret") pod "global-pull-secret-syncer-hs2qb" (UID: "9f8db869-87b0-4a2c-b292-b78f22f3a391") : object "kube-system"/"original-pull-secret" not registered Jul 26 16:08:49.577102 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:49.577056 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjgxp\" (UniqueName: \"kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp\") pod \"network-check-target-dndsz\" (UID: \"1d66c1fe-74b0-44a3-b892-bd68dd3ffc51\") " pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:08:49.577320 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:49.577264 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 16:08:49.577320 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:49.577296 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 16:08:49.577320 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:49.577309 2563 projected.go:196] Error preparing data for projected volume kube-api-access-fjgxp for pod openshift-network-diagnostics/network-check-target-dndsz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 16:08:49.577480 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:49.577385 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp podName:1d66c1fe-74b0-44a3-b892-bd68dd3ffc51 nodeName:}" failed. No retries permitted until 2026-07-26 16:08:50.577360661 +0000 UTC m=+4.158447230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fjgxp" (UniqueName: "kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp") pod "network-check-target-dndsz" (UID: "1d66c1fe-74b0-44a3-b892-bd68dd3ffc51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 16:08:49.799350 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:49.799295 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-25 16:03:48 +0000 UTC" deadline="2028-04-13 21:14:43.736950736 +0000 UTC" Jul 26 16:08:49.799350 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:49.799340 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15053h5m53.937615958s" Jul 26 16:08:49.885246 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:49.885206 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:08:49.885447 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:49.885357 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dndsz" podUID="1d66c1fe-74b0-44a3-b892-bd68dd3ffc51" Jul 26 16:08:50.028072 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:50.028040 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665e14a9_02c7_45b1_87d2_d9e98e559d12.slice/crio-b75a27ffdaf800e19298698b41e7cc2fff61fede546690affe1850aff8831a7d WatchSource:0}: Error finding container b75a27ffdaf800e19298698b41e7cc2fff61fede546690affe1850aff8831a7d: Status 404 returned error can't find the container with id b75a27ffdaf800e19298698b41e7cc2fff61fede546690affe1850aff8831a7d Jul 26 16:08:50.029824 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:50.029798 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456b66bc_bd4c_45da_96d6_6d1debc7987e.slice/crio-dc40037c5c20bfa140ce130c489eeeac62b321771033a35b9396b7e4afe10383 WatchSource:0}: Error finding container dc40037c5c20bfa140ce130c489eeeac62b321771033a35b9396b7e4afe10383: Status 404 returned error can't find the container with id dc40037c5c20bfa140ce130c489eeeac62b321771033a35b9396b7e4afe10383 Jul 26 16:08:50.037237 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:50.037204 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9ae82e_3ce4_4c7f_9902_a1ef79faae1f.slice/crio-1dc39ebb089c68ad43047c5a6a468a6ec8543d17138625eb63b27d0c600bc61f WatchSource:0}: Error finding container 1dc39ebb089c68ad43047c5a6a468a6ec8543d17138625eb63b27d0c600bc61f: Status 404 returned error can't find the container with id 1dc39ebb089c68ad43047c5a6a468a6ec8543d17138625eb63b27d0c600bc61f Jul 26 16:08:50.039640 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:08:50.039501 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fd86d4_e511_4f8e_962d_8c6726ba4cd1.slice/crio-74caeddbc11b805de2884f0b458cdae0525b82fb81547326e2910a3b09efe701 WatchSource:0}: Error finding container 74caeddbc11b805de2884f0b458cdae0525b82fb81547326e2910a3b09efe701: Status 404 returned error can't find the container with id 74caeddbc11b805de2884f0b458cdae0525b82fb81547326e2910a3b09efe701 Jul 26 16:08:50.383868 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:50.383599 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs\") pod \"network-metrics-daemon-qd8ng\" (UID: \"bc7a4aef-1953-45ab-84db-8338395b0b25\") " pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:08:50.383868 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:50.383765 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 16:08:50.384046 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:50.383896 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs podName:bc7a4aef-1953-45ab-84db-8338395b0b25 nodeName:}" failed. No retries permitted until 2026-07-26 16:08:52.383878129 +0000 UTC m=+5.964964680 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs") pod "network-metrics-daemon-qd8ng" (UID: "bc7a4aef-1953-45ab-84db-8338395b0b25") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 16:08:50.425029 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:50.425000 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 16:08:50.484905 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:50.484864 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret\") pod \"global-pull-secret-syncer-hs2qb\" (UID: \"9f8db869-87b0-4a2c-b292-b78f22f3a391\") " pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:08:50.485098 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:50.485080 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 16:08:50.485170 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:50.485158 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret podName:9f8db869-87b0-4a2c-b292-b78f22f3a391 nodeName:}" failed. No retries permitted until 2026-07-26 16:08:52.485138165 +0000 UTC m=+6.066224731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret") pod "global-pull-secret-syncer-hs2qb" (UID: "9f8db869-87b0-4a2c-b292-b78f22f3a391") : object "kube-system"/"original-pull-secret" not registered Jul 26 16:08:50.585740 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:50.585701 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjgxp\" (UniqueName: \"kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp\") pod \"network-check-target-dndsz\" (UID: \"1d66c1fe-74b0-44a3-b892-bd68dd3ffc51\") " pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:08:50.585952 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:50.585898 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 16:08:50.585952 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:50.585918 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 16:08:50.585952 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:50.585931 2563 projected.go:196] Error preparing data for projected volume kube-api-access-fjgxp for pod openshift-network-diagnostics/network-check-target-dndsz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 16:08:50.586128 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:50.585995 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp podName:1d66c1fe-74b0-44a3-b892-bd68dd3ffc51 nodeName:}" failed. No retries permitted until 2026-07-26 16:08:52.585974083 +0000 UTC m=+6.167060651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fjgxp" (UniqueName: "kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp") pod "network-check-target-dndsz" (UID: "1d66c1fe-74b0-44a3-b892-bd68dd3ffc51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 16:08:50.800367 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:50.800272 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-25 16:03:48 +0000 UTC" deadline="2028-03-17 06:01:19.556805818 +0000 UTC" Jul 26 16:08:50.800367 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:50.800337 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14389h52m28.756473423s" Jul 26 16:08:50.887174 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:50.887138 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:08:50.887389 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:50.887310 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8ng" podUID="bc7a4aef-1953-45ab-84db-8338395b0b25" Jul 26 16:08:50.887772 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:50.887739 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:08:50.887875 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:50.887853 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hs2qb" podUID="9f8db869-87b0-4a2c-b292-b78f22f3a391" Jul 26 16:08:50.927274 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:50.926836 2563 generic.go:356] "Generic (PLEG): container finished" podID="f917223d7503166a38a92a8b80122252" containerID="fea4747efb071c495cc84c0dcc0310ee37f842a762f11c4a49fca4d2058d820f" exitCode=0 Jul 26 16:08:50.927274 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:50.926936 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal" event={"ID":"f917223d7503166a38a92a8b80122252","Type":"ContainerDied","Data":"fea4747efb071c495cc84c0dcc0310ee37f842a762f11c4a49fca4d2058d820f"} Jul 26 16:08:50.941806 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:50.940359 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-knl4q" event={"ID":"d9ec0a41-3fb0-4b8e-a957-5a300790d89b","Type":"ContainerStarted","Data":"e2959e4f88e0eff26c42d19e74dcfe934fd797358bf18d5fa4a1b0085511d7f3"} Jul 26 16:08:50.946437 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:50.946373 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l7cw2" event={"ID":"f7f65fa5-91ab-49cf-ab27-f4af947b3bbe","Type":"ContainerStarted","Data":"464cfde24f92406250bd36f4c49fed926517506130625e40d74fd239480a3343"} Jul 26 16:08:50.953908 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:50.953821 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2nwxs" event={"ID":"e8fd86d4-e511-4f8e-962d-8c6726ba4cd1","Type":"ContainerStarted","Data":"74caeddbc11b805de2884f0b458cdae0525b82fb81547326e2910a3b09efe701"} Jul 26 16:08:50.976417 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:50.976336 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bqnx2" event={"ID":"8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f","Type":"ContainerStarted","Data":"1dc39ebb089c68ad43047c5a6a468a6ec8543d17138625eb63b27d0c600bc61f"} Jul 26 16:08:50.994206 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:50.994174 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" event={"ID":"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3","Type":"ContainerStarted","Data":"855cec7eb8c12ec370adff127e17149cca46b04f217f9d9e6cabd6ebcfe0739b"} Jul 26 16:08:51.023375 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:51.023338 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" event={"ID":"456b66bc-bd4c-45da-96d6-6d1debc7987e","Type":"ContainerStarted","Data":"dc40037c5c20bfa140ce130c489eeeac62b321771033a35b9396b7e4afe10383"} Jul 26 16:08:51.040063 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:51.039448 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-70.ec2.internal" event={"ID":"e33f5784a8667eeea92ef51af7a40e20","Type":"ContainerStarted","Data":"9500cea3fc29ce2c2af605d5b536f6644261ce8b86ae916daa8ac14b6a003685"} Jul 26 16:08:51.048958 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:51.048926 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bd5xf" event={"ID":"d83251cf-1af1-438a-9525-a79558f5f48f","Type":"ContainerStarted","Data":"f61289c9b9592a1952991fcc38a0251001cfb585f0f35afa312e3d2229dfdf0c"} Jul 26 16:08:51.055371 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:51.055274 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-70.ec2.internal" podStartSLOduration=3.055255414 podStartE2EDuration="3.055255414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 16:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:08:51.054587445 +0000 UTC m=+4.635674017" watchObservedRunningTime="2026-07-26 16:08:51.055255414 +0000 UTC m=+4.636341987" Jul 26 16:08:51.064078 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:51.063976 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m98wg" event={"ID":"32afde23-1522-43c3-93dc-fd4a60a27d5f","Type":"ContainerStarted","Data":"6551862e755cb99606b35534df82b49b0a8d0c60edc82d6d8ed23f8a09eca302"} Jul 26 16:08:51.077587 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:51.077549 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" event={"ID":"665e14a9-02c7-45b1-87d2-d9e98e559d12","Type":"ContainerStarted","Data":"b75a27ffdaf800e19298698b41e7cc2fff61fede546690affe1850aff8831a7d"} Jul 26 16:08:51.884900 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:51.884864 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:08:51.885495 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:51.884998 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dndsz" podUID="1d66c1fe-74b0-44a3-b892-bd68dd3ffc51" Jul 26 16:08:52.100246 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:52.100066 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal" event={"ID":"f917223d7503166a38a92a8b80122252","Type":"ContainerStarted","Data":"793c2d6f4d7ce9c83bd991c54a33aa99a52cb17dfe269cb77696ad186e39aed8"} Jul 26 16:08:52.400290 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:52.400247 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs\") pod \"network-metrics-daemon-qd8ng\" (UID: \"bc7a4aef-1953-45ab-84db-8338395b0b25\") " pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:08:52.400481 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:52.400392 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 16:08:52.400481 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:52.400456 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs podName:bc7a4aef-1953-45ab-84db-8338395b0b25 nodeName:}" failed. No retries permitted until 2026-07-26 16:08:56.400436793 +0000 UTC m=+9.981523344 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs") pod "network-metrics-daemon-qd8ng" (UID: "bc7a4aef-1953-45ab-84db-8338395b0b25") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 16:08:52.501236 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:52.501181 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret\") pod \"global-pull-secret-syncer-hs2qb\" (UID: \"9f8db869-87b0-4a2c-b292-b78f22f3a391\") " pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:08:52.501440 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:52.501392 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 16:08:52.501510 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:52.501463 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret podName:9f8db869-87b0-4a2c-b292-b78f22f3a391 nodeName:}" failed. No retries permitted until 2026-07-26 16:08:56.501436548 +0000 UTC m=+10.082523119 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret") pod "global-pull-secret-syncer-hs2qb" (UID: "9f8db869-87b0-4a2c-b292-b78f22f3a391") : object "kube-system"/"original-pull-secret" not registered Jul 26 16:08:52.602570 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:52.602215 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjgxp\" (UniqueName: \"kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp\") pod \"network-check-target-dndsz\" (UID: \"1d66c1fe-74b0-44a3-b892-bd68dd3ffc51\") " pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:08:52.603266 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:52.602816 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 16:08:52.603266 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:52.602849 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 16:08:52.603266 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:52.602862 2563 projected.go:196] Error preparing data for projected volume kube-api-access-fjgxp for pod openshift-network-diagnostics/network-check-target-dndsz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 16:08:52.603266 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:52.602929 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp podName:1d66c1fe-74b0-44a3-b892-bd68dd3ffc51 nodeName:}" failed. No retries permitted until 2026-07-26 16:08:56.602909756 +0000 UTC m=+10.183996319 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fjgxp" (UniqueName: "kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp") pod "network-check-target-dndsz" (UID: "1d66c1fe-74b0-44a3-b892-bd68dd3ffc51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 16:08:52.887614 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:52.887581 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:08:52.888045 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:52.887705 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hs2qb" podUID="9f8db869-87b0-4a2c-b292-b78f22f3a391" Jul 26 16:08:52.888045 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:52.887819 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:08:52.888045 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:52.887949 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8ng" podUID="bc7a4aef-1953-45ab-84db-8338395b0b25" Jul 26 16:08:53.884968 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:53.884931 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:08:53.885201 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:53.885070 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dndsz" podUID="1d66c1fe-74b0-44a3-b892-bd68dd3ffc51" Jul 26 16:08:54.885365 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:54.885333 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:08:54.885365 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:54.885357 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:08:54.885879 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:54.885490 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8ng" podUID="bc7a4aef-1953-45ab-84db-8338395b0b25" Jul 26 16:08:54.885879 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:54.885646 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hs2qb" podUID="9f8db869-87b0-4a2c-b292-b78f22f3a391" Jul 26 16:08:55.885031 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:55.884991 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:08:55.885250 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:55.885129 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dndsz" podUID="1d66c1fe-74b0-44a3-b892-bd68dd3ffc51" Jul 26 16:08:56.433302 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:56.433267 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs\") pod \"network-metrics-daemon-qd8ng\" (UID: \"bc7a4aef-1953-45ab-84db-8338395b0b25\") " pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:08:56.433759 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:56.433401 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 16:08:56.433759 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:56.433466 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs podName:bc7a4aef-1953-45ab-84db-8338395b0b25 nodeName:}" failed. No retries permitted until 2026-07-26 16:09:04.433438787 +0000 UTC m=+18.014525351 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs") pod "network-metrics-daemon-qd8ng" (UID: "bc7a4aef-1953-45ab-84db-8338395b0b25") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 16:08:56.533771 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:56.533725 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret\") pod \"global-pull-secret-syncer-hs2qb\" (UID: \"9f8db869-87b0-4a2c-b292-b78f22f3a391\") " pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:08:56.533962 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:56.533894 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 16:08:56.534029 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:56.533979 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret podName:9f8db869-87b0-4a2c-b292-b78f22f3a391 nodeName:}" failed. No retries permitted until 2026-07-26 16:09:04.533956888 +0000 UTC m=+18.115043450 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret") pod "global-pull-secret-syncer-hs2qb" (UID: "9f8db869-87b0-4a2c-b292-b78f22f3a391") : object "kube-system"/"original-pull-secret" not registered Jul 26 16:08:56.635178 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:56.635135 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjgxp\" (UniqueName: \"kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp\") pod \"network-check-target-dndsz\" (UID: \"1d66c1fe-74b0-44a3-b892-bd68dd3ffc51\") " pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:08:56.635353 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:56.635303 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 16:08:56.635353 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:56.635319 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 16:08:56.635353 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:56.635328 2563 projected.go:196] Error preparing data for projected volume kube-api-access-fjgxp for pod openshift-network-diagnostics/network-check-target-dndsz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 16:08:56.635525 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:56.635389 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp podName:1d66c1fe-74b0-44a3-b892-bd68dd3ffc51 nodeName:}" failed. No retries permitted until 2026-07-26 16:09:04.635370416 +0000 UTC m=+18.216456970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fjgxp" (UniqueName: "kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp") pod "network-check-target-dndsz" (UID: "1d66c1fe-74b0-44a3-b892-bd68dd3ffc51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 16:08:56.886302 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:56.886266 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:08:56.886498 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:56.886399 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8ng" podUID="bc7a4aef-1953-45ab-84db-8338395b0b25" Jul 26 16:08:56.886869 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:56.886730 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:08:56.886869 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:56.886828 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hs2qb" podUID="9f8db869-87b0-4a2c-b292-b78f22f3a391" Jul 26 16:08:57.885123 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:57.885086 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:08:57.885626 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:57.885236 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dndsz" podUID="1d66c1fe-74b0-44a3-b892-bd68dd3ffc51" Jul 26 16:08:58.884888 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:58.884802 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:08:58.884888 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:58.884836 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:08:58.885072 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:58.884962 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8ng" podUID="bc7a4aef-1953-45ab-84db-8338395b0b25" Jul 26 16:08:58.885362 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:58.885318 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hs2qb" podUID="9f8db869-87b0-4a2c-b292-b78f22f3a391" Jul 26 16:08:59.884647 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:08:59.884610 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:08:59.884835 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:08:59.884734 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dndsz" podUID="1d66c1fe-74b0-44a3-b892-bd68dd3ffc51" Jul 26 16:09:00.884982 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:00.884944 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:09:00.885491 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:00.885091 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8ng" podUID="bc7a4aef-1953-45ab-84db-8338395b0b25" Jul 26 16:09:00.885491 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:00.885198 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:09:00.885491 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:00.885323 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hs2qb" podUID="9f8db869-87b0-4a2c-b292-b78f22f3a391" Jul 26 16:09:01.884433 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:01.884397 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:09:01.884593 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:01.884519 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dndsz" podUID="1d66c1fe-74b0-44a3-b892-bd68dd3ffc51" Jul 26 16:09:02.885207 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:02.885169 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:09:02.885706 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:02.885169 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:09:02.885706 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:02.885333 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8ng" podUID="bc7a4aef-1953-45ab-84db-8338395b0b25" Jul 26 16:09:02.885706 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:02.885400 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hs2qb" podUID="9f8db869-87b0-4a2c-b292-b78f22f3a391" Jul 26 16:09:03.885208 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:03.885177 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:09:03.885675 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:03.885299 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dndsz" podUID="1d66c1fe-74b0-44a3-b892-bd68dd3ffc51" Jul 26 16:09:04.499360 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:04.499282 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs\") pod \"network-metrics-daemon-qd8ng\" (UID: \"bc7a4aef-1953-45ab-84db-8338395b0b25\") " pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:09:04.499563 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:04.499399 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 16:09:04.499563 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:04.499453 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs podName:bc7a4aef-1953-45ab-84db-8338395b0b25 nodeName:}" failed. No retries permitted until 2026-07-26 16:09:20.499437869 +0000 UTC m=+34.080524417 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs") pod "network-metrics-daemon-qd8ng" (UID: "bc7a4aef-1953-45ab-84db-8338395b0b25") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 16:09:04.600147 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:04.600103 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret\") pod \"global-pull-secret-syncer-hs2qb\" (UID: \"9f8db869-87b0-4a2c-b292-b78f22f3a391\") " pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:09:04.600343 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:04.600268 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 16:09:04.600343 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:04.600339 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret podName:9f8db869-87b0-4a2c-b292-b78f22f3a391 nodeName:}" failed. No retries permitted until 2026-07-26 16:09:20.600321724 +0000 UTC m=+34.181408273 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret") pod "global-pull-secret-syncer-hs2qb" (UID: "9f8db869-87b0-4a2c-b292-b78f22f3a391") : object "kube-system"/"original-pull-secret" not registered Jul 26 16:09:04.700867 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:04.700833 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjgxp\" (UniqueName: \"kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp\") pod \"network-check-target-dndsz\" (UID: \"1d66c1fe-74b0-44a3-b892-bd68dd3ffc51\") " pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:09:04.701027 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:04.701009 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 16:09:04.701071 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:04.701031 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 16:09:04.701071 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:04.701041 2563 projected.go:196] Error preparing data for projected volume kube-api-access-fjgxp for pod openshift-network-diagnostics/network-check-target-dndsz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 16:09:04.701146 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:04.701101 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp podName:1d66c1fe-74b0-44a3-b892-bd68dd3ffc51 nodeName:}" failed. No retries permitted until 2026-07-26 16:09:20.70108253 +0000 UTC m=+34.282169103 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fjgxp" (UniqueName: "kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp") pod "network-check-target-dndsz" (UID: "1d66c1fe-74b0-44a3-b892-bd68dd3ffc51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 16:09:04.885145 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:04.885106 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:09:04.885339 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:04.885106 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:09:04.885339 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:04.885269 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8ng" podUID="bc7a4aef-1953-45ab-84db-8338395b0b25" Jul 26 16:09:04.885696 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:04.885336 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hs2qb" podUID="9f8db869-87b0-4a2c-b292-b78f22f3a391" Jul 26 16:09:05.884462 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:05.884420 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:09:05.884646 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:05.884545 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dndsz" podUID="1d66c1fe-74b0-44a3-b892-bd68dd3ffc51" Jul 26 16:09:06.885732 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:06.885684 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:09:06.886180 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:06.885696 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:09:06.886180 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:06.885816 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8ng" podUID="bc7a4aef-1953-45ab-84db-8338395b0b25" Jul 26 16:09:06.886180 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:06.885912 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hs2qb" podUID="9f8db869-87b0-4a2c-b292-b78f22f3a391" Jul 26 16:09:07.884361 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:07.884325 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:09:07.884558 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:07.884436 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dndsz" podUID="1d66c1fe-74b0-44a3-b892-bd68dd3ffc51" Jul 26 16:09:08.885525 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:08.885384 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:09:08.886372 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:08.885610 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hs2qb" podUID="9f8db869-87b0-4a2c-b292-b78f22f3a391" Jul 26 16:09:08.886372 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:08.885867 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:09:08.886372 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:08.885940 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8ng" podUID="bc7a4aef-1953-45ab-84db-8338395b0b25" Jul 26 16:09:09.141534 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:09.141310 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bqnx2" event={"ID":"8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f","Type":"ContainerStarted","Data":"88e0749fec9a8843a77cbd47ffbeeb912166a9acaa715983c9177812082cd58c"} Jul 26 16:09:09.143326 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:09.143305 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" event={"ID":"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3","Type":"ContainerStarted","Data":"91615938e770982e8385a1ee721326d4aeb8014b5c5bc113a8e3550d0f4ea39c"} Jul 26 16:09:09.143434 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:09.143379 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" event={"ID":"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3","Type":"ContainerStarted","Data":"32bec4699b8c5ce5d1a214d3d6936d0035ce01cead3302211904e8f143611500"} Jul 26 16:09:09.143498 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:09.143432 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" event={"ID":"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3","Type":"ContainerStarted","Data":"bab3d205a6dd0a38e56b30e75598b7299b788ab3e61b1632053289a622e6158f"} Jul 26 16:09:09.144641 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:09.144608 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" event={"ID":"456b66bc-bd4c-45da-96d6-6d1debc7987e","Type":"ContainerStarted","Data":"24dda6b0c56b0818f98fd441e6505e1919f18ab20fc91a1c269a9816227097cc"} Jul 26 16:09:09.146149 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:09.146127 2563 generic.go:356] "Generic (PLEG): container finished" podID="d83251cf-1af1-438a-9525-a79558f5f48f" containerID="6fb1767fda9e3016eadc752bbb711b623ccc8b73f66ba098590f0b024f0f958e" exitCode=0 Jul 26 16:09:09.146250 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:09.146208 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bd5xf" event={"ID":"d83251cf-1af1-438a-9525-a79558f5f48f","Type":"ContainerDied","Data":"6fb1767fda9e3016eadc752bbb711b623ccc8b73f66ba098590f0b024f0f958e"} Jul 26 16:09:09.147885 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:09.147822 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" event={"ID":"665e14a9-02c7-45b1-87d2-d9e98e559d12","Type":"ContainerStarted","Data":"420d5885014a034553dcdadc243c7b9e21a4aceccdbd4159943d45ee93c23435"} Jul 26 16:09:09.149651 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:09.149624 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-knl4q" event={"ID":"d9ec0a41-3fb0-4b8e-a957-5a300790d89b","Type":"ContainerStarted","Data":"9c95dddb80507bb626c88ee5eb25158a13f9019fe7bf34392eed6ce8f8b35c80"} Jul 26 16:09:09.151054 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:09.151030 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l7cw2" event={"ID":"f7f65fa5-91ab-49cf-ab27-f4af947b3bbe","Type":"ContainerStarted","Data":"ac5d7fa8a7550e2d047a2e922ec70dc72654591953afd6e0fa49619013fa4b14"} Jul 26 16:09:09.152463 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:09.152443 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2nwxs" event={"ID":"e8fd86d4-e511-4f8e-962d-8c6726ba4cd1","Type":"ContainerStarted","Data":"8b92764e06b3150e89b17a785561209e1430a8d33e282b934685334930071d9f"} Jul 26 16:09:09.156373 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:09.156337 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bqnx2" podStartSLOduration=3.580429741 podStartE2EDuration="22.156326337s" totalImagesPullingTime="18.575896596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 16:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:09.156138118 +0000 UTC m=+22.737224689" watchObservedRunningTime="2026-07-26 16:09:09.156326337 +0000 UTC m=+22.737412906" Jul 26 16:09:09.156447 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:09.156432 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-70.ec2.internal" podStartSLOduration=21.156428058 podStartE2EDuration="21.156428058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 16:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:08:52.117662303 +0000 UTC m=+5.698748878" watchObservedRunningTime="2026-07-26 16:09:09.156428058 +0000 UTC m=+22.737514630" Jul 26 16:09:09.172884 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:09.172844 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-knl4q" podStartSLOduration=3.543162811 podStartE2EDuration="22.17283147s" totalImagesPullingTime="18.629668659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 16:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:09.172671364 +0000 UTC m=+22.753757934" watchObservedRunningTime="2026-07-26 16:09:09.17283147 +0000 UTC m=+22.753918040" Jul 26 16:09:09.188957 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:09.188913 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2nwxs" podStartSLOduration=3.620738636 podStartE2EDuration="22.188899395s" totalImagesPullingTime="18.568160759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 16:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:09.188677781 +0000 UTC m=+22.769764352" watchObservedRunningTime="2026-07-26 16:09:09.188899395 +0000 UTC m=+22.769985965" Jul 26 16:09:09.203720 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:09.203684 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l7cw2" podStartSLOduration=8.097200296 podStartE2EDuration="22.203665802s" totalImagesPullingTime="14.106465506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:09.203463594 +0000 UTC m=+22.784550163" watchObservedRunningTime="2026-07-26 16:09:09.203665802 +0000 UTC m=+22.784752349" Jul 26 16:09:09.219259 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:09.219194 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mt7v8" podStartSLOduration=3.609623455 podStartE2EDuration="22.219175003s" totalImagesPullingTime="18.609551548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 16:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:09.218967276 +0000 UTC m=+22.800053847" watchObservedRunningTime="2026-07-26 16:09:09.219175003 +0000 UTC m=+22.800261574" Jul 26 16:09:09.884425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:09.884395 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:09:09.884685 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:09.884504 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dndsz" podUID="1d66c1fe-74b0-44a3-b892-bd68dd3ffc51" Jul 26 16:09:10.074910 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:10.074876 2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 26 16:09:10.156530 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:10.156445 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m98wg" event={"ID":"32afde23-1522-43c3-93dc-fd4a60a27d5f","Type":"ContainerStarted","Data":"421f87aea77c135fdbd728e648c6ef5d7e4ed95c91fe2e76e7a11cad5aaaf51f"} Jul 26 16:09:10.163650 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:10.163623 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" event={"ID":"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3","Type":"ContainerStarted","Data":"95b62f0a6fbef154648425c78058eb189163831fae59c5c10e15a7f207ad4c1b"} Jul 26 16:09:10.163837 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:10.163737 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" event={"ID":"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3","Type":"ContainerStarted","Data":"14610ce027ae9708b7ed9e339e74f809517ce4ae357a2d1d6f5f196c1ac93d30"} Jul 26 16:09:10.163837 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:10.163793 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" event={"ID":"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3","Type":"ContainerStarted","Data":"e20ef09b099f0f2d22de9451d3c4de11698d7cbfe2a08b587c5a5f611fdf6cd3"} Jul 26 16:09:10.165648 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:10.165544 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" event={"ID":"456b66bc-bd4c-45da-96d6-6d1debc7987e","Type":"ContainerStarted","Data":"36a2409caf26d41d1aa83a7606890c865dce2d958378759f179a83d1b04d3f99"} Jul 26 16:09:10.171625 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:10.171581 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m98wg" podStartSLOduration=4.605913072 podStartE2EDuration="23.171563505s" totalImagesPullingTime="18.565650433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 16:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:10.171372024 +0000 UTC m=+23.752458596" watchObservedRunningTime="2026-07-26 16:09:10.171563505 +0000 UTC m=+23.752650076" Jul 26 16:09:10.569515 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:10.569481 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l7cw2" Jul 26 16:09:10.570047 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:10.570024 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l7cw2" Jul 26 16:09:10.824726 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:10.824530 2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-26T16:09:10.074907191Z","UUID":"d4733805-bdd5-4872-8caf-eb01470d662c","Handler":null,"Name":"","Endpoint":""} Jul 26 16:09:10.826250 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:10.826212 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 26 16:09:10.826372 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:10.826276 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 26 16:09:10.884717 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:10.884688 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:09:10.884877 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:10.884803 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hs2qb" podUID="9f8db869-87b0-4a2c-b292-b78f22f3a391" Jul 26 16:09:10.885147 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:10.884689 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:09:10.885316 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:10.885269 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8ng" podUID="bc7a4aef-1953-45ab-84db-8338395b0b25" Jul 26 16:09:11.167879 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:11.167852 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l7cw2" Jul 26 16:09:11.168456 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:11.168427 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l7cw2" Jul 26 16:09:11.884693 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:11.884663 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:09:11.884959 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:11.884787 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dndsz" podUID="1d66c1fe-74b0-44a3-b892-bd68dd3ffc51" Jul 26 16:09:12.173511 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:12.173430 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" event={"ID":"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3","Type":"ContainerStarted","Data":"4e3b116d8aeeaf643a018aea31c7bd6fed48951b4816b6151d78f6b88c395062"} Jul 26 16:09:12.175694 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:12.175668 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" event={"ID":"456b66bc-bd4c-45da-96d6-6d1debc7987e","Type":"ContainerStarted","Data":"546ad12ccb25e943edb9d4ad95c5789f2257300a2427aba3bcb53a5c9176418e"} Jul 26 16:09:12.194152 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:12.194106 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4jjvl" podStartSLOduration=4.446289854 podStartE2EDuration="25.194091668s" totalImagesPullingTime="20.747801814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 16:08:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:12.192531072 +0000 UTC m=+25.773617642" watchObservedRunningTime="2026-07-26 16:09:12.194091668 +0000 UTC m=+25.775178250" Jul 26 16:09:12.884996 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:12.884961 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:09:12.885172 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:12.884961 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:09:12.885172 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:12.885100 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hs2qb" podUID="9f8db869-87b0-4a2c-b292-b78f22f3a391" Jul 26 16:09:12.885172 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:12.885147 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8ng" podUID="bc7a4aef-1953-45ab-84db-8338395b0b25" Jul 26 16:09:13.884694 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:13.884528 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:09:13.885328 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:13.884773 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dndsz" podUID="1d66c1fe-74b0-44a3-b892-bd68dd3ffc51" Jul 26 16:09:14.182274 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:14.182186 2563 generic.go:356] "Generic (PLEG): container finished" podID="d83251cf-1af1-438a-9525-a79558f5f48f" containerID="48f37ddc1a314d16c58c93256dd56d9a07caea438a4bf4b5ddc04c130c96a38c" exitCode=0 Jul 26 16:09:14.182274 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:14.182261 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bd5xf" event={"ID":"d83251cf-1af1-438a-9525-a79558f5f48f","Type":"ContainerDied","Data":"48f37ddc1a314d16c58c93256dd56d9a07caea438a4bf4b5ddc04c130c96a38c"} Jul 26 16:09:14.186133 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:14.186111 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" event={"ID":"7ba16aca-ddb5-4c58-a2a5-89d8b52150a3","Type":"ContainerStarted","Data":"da568959e8b8548f086405eb802baad9a8666dea7c935ebd409ff23205dad241"} Jul 26 16:09:14.186454 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:14.186434 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:09:14.186539 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:14.186511 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:09:14.201563 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:14.201543 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:09:14.272692 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:14.272647 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" podStartSLOduration=8.623508624 podStartE2EDuration="27.272632662s" totalImagesPullingTime="18.649124038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 16:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:14.255031861 +0000 UTC m=+27.836118430" watchObservedRunningTime="2026-07-26 16:09:14.272632662 +0000 UTC m=+27.853719231" Jul 26 16:09:14.884988 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:14.884827 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:09:14.885428 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:14.884830 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:09:14.885428 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:14.885095 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hs2qb" podUID="9f8db869-87b0-4a2c-b292-b78f22f3a391" Jul 26 16:09:14.885428 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:14.885148 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8ng" podUID="bc7a4aef-1953-45ab-84db-8338395b0b25" Jul 26 16:09:15.188478 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:15.188390 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:09:15.203357 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:15.203331 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:09:15.707921 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:15.707832 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dndsz"] Jul 26 16:09:15.708115 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:15.708004 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:09:15.708186 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:15.708115 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dndsz" podUID="1d66c1fe-74b0-44a3-b892-bd68dd3ffc51" Jul 26 16:09:15.708682 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:15.708657 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qd8ng"] Jul 26 16:09:15.708819 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:15.708765 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:09:15.708892 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:15.708871 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8ng" podUID="bc7a4aef-1953-45ab-84db-8338395b0b25" Jul 26 16:09:15.709281 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:15.709261 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hs2qb"] Jul 26 16:09:15.709368 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:15.709342 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:09:15.709452 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:15.709421 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hs2qb" podUID="9f8db869-87b0-4a2c-b292-b78f22f3a391" Jul 26 16:09:16.192781 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:16.192748 2563 generic.go:356] "Generic (PLEG): container finished" podID="d83251cf-1af1-438a-9525-a79558f5f48f" containerID="b68ee100555ce0afd85818a5e93f015217722e0ee27266df8c53becfc182b22f" exitCode=0 Jul 26 16:09:16.193194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:16.192838 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bd5xf" event={"ID":"d83251cf-1af1-438a-9525-a79558f5f48f","Type":"ContainerDied","Data":"b68ee100555ce0afd85818a5e93f015217722e0ee27266df8c53becfc182b22f"} Jul 26 16:09:16.885978 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:16.885940 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:09:16.886134 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:16.886075 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8ng" podUID="bc7a4aef-1953-45ab-84db-8338395b0b25" Jul 26 16:09:17.885302 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:17.885111 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:09:17.885683 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:17.885111 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:09:17.885683 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:17.885397 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dndsz" podUID="1d66c1fe-74b0-44a3-b892-bd68dd3ffc51" Jul 26 16:09:17.885683 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:17.885450 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hs2qb" podUID="9f8db869-87b0-4a2c-b292-b78f22f3a391" Jul 26 16:09:18.200364 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:18.200331 2563 generic.go:356] "Generic (PLEG): container finished" podID="d83251cf-1af1-438a-9525-a79558f5f48f" containerID="65a2f4472829eaad06213e90c5682bed29ee8cc484bf4b5177eaa442edbfe1ee" exitCode=0 Jul 26 16:09:18.200525 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:18.200375 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bd5xf" event={"ID":"d83251cf-1af1-438a-9525-a79558f5f48f","Type":"ContainerDied","Data":"65a2f4472829eaad06213e90c5682bed29ee8cc484bf4b5177eaa442edbfe1ee"} Jul 26 16:09:18.884872 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:18.884837 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:09:18.885049 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:18.884984 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8ng" podUID="bc7a4aef-1953-45ab-84db-8338395b0b25" Jul 26 16:09:19.884686 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:19.884649 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:09:19.885111 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:19.884649 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:09:19.885111 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:19.884758 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hs2qb" podUID="9f8db869-87b0-4a2c-b292-b78f22f3a391" Jul 26 16:09:19.885111 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:19.884860 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dndsz" podUID="1d66c1fe-74b0-44a3-b892-bd68dd3ffc51" Jul 26 16:09:20.520619 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:20.520584 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs\") pod \"network-metrics-daemon-qd8ng\" (UID: \"bc7a4aef-1953-45ab-84db-8338395b0b25\") " pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:09:20.520834 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:20.520731 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 16:09:20.520834 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:20.520803 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs podName:bc7a4aef-1953-45ab-84db-8338395b0b25 nodeName:}" failed. No retries permitted until 2026-07-26 16:09:52.520783497 +0000 UTC m=+66.101870062 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs") pod "network-metrics-daemon-qd8ng" (UID: "bc7a4aef-1953-45ab-84db-8338395b0b25") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 16:09:20.621614 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:20.621580 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret\") pod \"global-pull-secret-syncer-hs2qb\" (UID: \"9f8db869-87b0-4a2c-b292-b78f22f3a391\") " pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:09:20.621771 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:20.621707 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 16:09:20.621771 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:20.621761 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret podName:9f8db869-87b0-4a2c-b292-b78f22f3a391 nodeName:}" failed. No retries permitted until 2026-07-26 16:09:52.621748004 +0000 UTC m=+66.202834552 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret") pod "global-pull-secret-syncer-hs2qb" (UID: "9f8db869-87b0-4a2c-b292-b78f22f3a391") : object "kube-system"/"original-pull-secret" not registered Jul 26 16:09:20.722709 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:20.722670 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjgxp\" (UniqueName: \"kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp\") pod \"network-check-target-dndsz\" (UID: \"1d66c1fe-74b0-44a3-b892-bd68dd3ffc51\") " pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:09:20.722898 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:20.722856 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 16:09:20.722898 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:20.722885 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 16:09:20.722898 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:20.722899 2563 projected.go:196] Error preparing data for projected volume kube-api-access-fjgxp for pod openshift-network-diagnostics/network-check-target-dndsz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 16:09:20.723057 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:20.722958 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp podName:1d66c1fe-74b0-44a3-b892-bd68dd3ffc51 nodeName:}" failed. No retries permitted until 2026-07-26 16:09:52.72294019 +0000 UTC m=+66.304026741 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fjgxp" (UniqueName: "kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp") pod "network-check-target-dndsz" (UID: "1d66c1fe-74b0-44a3-b892-bd68dd3ffc51") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 16:09:20.884691 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:20.884661 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:09:20.885102 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:20.884805 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8ng" podUID="bc7a4aef-1953-45ab-84db-8338395b0b25" Jul 26 16:09:21.256703 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.256667 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-70.ec2.internal" event="NodeReady" Jul 26 16:09:21.256876 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.256817 2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 26 16:09:21.301371 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.301338 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b27r6"] Jul 26 16:09:21.303856 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.303827 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hsc48"] Jul 26 16:09:21.303991 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.303977 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b27r6" Jul 26 16:09:21.305763 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.305738 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hsc48" Jul 26 16:09:21.307030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.307003 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 26 16:09:21.307191 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.307166 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-grxdp\"" Jul 26 16:09:21.307303 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.307209 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 26 16:09:21.309708 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.309688 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 26 16:09:21.309930 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.309913 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vf2c6\"" Jul 26 16:09:21.310019 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.309959 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 26 16:09:21.310215 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.310198 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 26 16:09:21.315543 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.315522 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b27r6"] Jul 26 16:09:21.319479 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.319422 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hsc48"] Jul 26 16:09:21.319615 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.319589 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-599f4f87f4-25xgn"] Jul 26 16:09:21.321671 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.321560 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.324341 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.324202 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 26 16:09:21.324341 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.324213 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 26 16:09:21.324341 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.324202 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 26 16:09:21.324341 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.324262 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 26 16:09:21.324602 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.324360 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 26 16:09:21.324602 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.324507 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hkc8g\"" Jul 26 16:09:21.324602 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.324582 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 26 16:09:21.324745 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.324610 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 26 16:09:21.330011 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.329993 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2sr94"] Jul 26 16:09:21.331918 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.331901 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2sr94" Jul 26 16:09:21.333940 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.333916 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-599f4f87f4-25xgn"] Jul 26 16:09:21.334410 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.334390 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r6bm5\"" Jul 26 16:09:21.334490 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.334412 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 26 16:09:21.334560 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.334543 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 26 16:09:21.334661 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.334645 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 26 16:09:21.335645 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.335626 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 26 16:09:21.344820 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.344800 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2sr94"] Jul 26 16:09:21.429370 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.429289 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/358697ed-6e24-42e2-9687-eee030fe181c-console-serving-cert\") pod \"console-599f4f87f4-25xgn\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.429370 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.429333 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/358697ed-6e24-42e2-9687-eee030fe181c-oauth-serving-cert\") pod \"console-599f4f87f4-25xgn\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.429370 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.429363 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2c98da9-5848-4b54-8034-d60ef7e81fcd-data-volume\") pod \"insights-runtime-extractor-2sr94\" (UID: \"c2c98da9-5848-4b54-8034-d60ef7e81fcd\") " pod="openshift-insights/insights-runtime-extractor-2sr94" Jul 26 16:09:21.429638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.429392 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfldh\" (UniqueName: \"kubernetes.io/projected/3cee3f58-a577-48df-9b42-ea4fba7ed3cf-kube-api-access-wfldh\") pod \"dns-default-b27r6\" (UID: \"3cee3f58-a577-48df-9b42-ea4fba7ed3cf\") " pod="openshift-dns/dns-default-b27r6" Jul 26 16:09:21.429638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.429423 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c2c98da9-5848-4b54-8034-d60ef7e81fcd-crio-socket\") pod \"insights-runtime-extractor-2sr94\" (UID: \"c2c98da9-5848-4b54-8034-d60ef7e81fcd\") " pod="openshift-insights/insights-runtime-extractor-2sr94" Jul 26 16:09:21.429638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.429457 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2c98da9-5848-4b54-8034-d60ef7e81fcd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2sr94\" (UID: \"c2c98da9-5848-4b54-8034-d60ef7e81fcd\") " pod="openshift-insights/insights-runtime-extractor-2sr94" Jul 26 16:09:21.429638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.429519 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55p6d\" (UniqueName: \"kubernetes.io/projected/62ca1095-ccae-444f-b8e6-2625663e191f-kube-api-access-55p6d\") pod \"ingress-canary-hsc48\" (UID: \"62ca1095-ccae-444f-b8e6-2625663e191f\") " pod="openshift-ingress-canary/ingress-canary-hsc48" Jul 26 16:09:21.429638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.429543 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/358697ed-6e24-42e2-9687-eee030fe181c-service-ca\") pod \"console-599f4f87f4-25xgn\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.429638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.429567 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62ca1095-ccae-444f-b8e6-2625663e191f-cert\") pod \"ingress-canary-hsc48\" (UID: \"62ca1095-ccae-444f-b8e6-2625663e191f\") " pod="openshift-ingress-canary/ingress-canary-hsc48" Jul 26 16:09:21.429638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.429590 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkqlh\" (UniqueName: \"kubernetes.io/projected/c2c98da9-5848-4b54-8034-d60ef7e81fcd-kube-api-access-jkqlh\") pod \"insights-runtime-extractor-2sr94\" (UID: \"c2c98da9-5848-4b54-8034-d60ef7e81fcd\") " pod="openshift-insights/insights-runtime-extractor-2sr94" Jul 26 16:09:21.429638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.429612 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cee3f58-a577-48df-9b42-ea4fba7ed3cf-tmp-dir\") pod \"dns-default-b27r6\" (UID: \"3cee3f58-a577-48df-9b42-ea4fba7ed3cf\") " pod="openshift-dns/dns-default-b27r6" Jul 26 16:09:21.429868 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.429644 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/358697ed-6e24-42e2-9687-eee030fe181c-console-oauth-config\") pod \"console-599f4f87f4-25xgn\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.429868 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.429685 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c2c98da9-5848-4b54-8034-d60ef7e81fcd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2sr94\" (UID: \"c2c98da9-5848-4b54-8034-d60ef7e81fcd\") " pod="openshift-insights/insights-runtime-extractor-2sr94" Jul 26 16:09:21.429868 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.429701 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3cee3f58-a577-48df-9b42-ea4fba7ed3cf-config-volume\") pod \"dns-default-b27r6\" (UID: \"3cee3f58-a577-48df-9b42-ea4fba7ed3cf\") " pod="openshift-dns/dns-default-b27r6" Jul 26 16:09:21.429868 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.429714 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwjnn\" (UniqueName: \"kubernetes.io/projected/358697ed-6e24-42e2-9687-eee030fe181c-kube-api-access-hwjnn\") pod \"console-599f4f87f4-25xgn\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.429868 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.429736 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cee3f58-a577-48df-9b42-ea4fba7ed3cf-metrics-tls\") pod \"dns-default-b27r6\" (UID: \"3cee3f58-a577-48df-9b42-ea4fba7ed3cf\") " pod="openshift-dns/dns-default-b27r6" Jul 26 16:09:21.429868 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.429749 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/358697ed-6e24-42e2-9687-eee030fe181c-console-config\") pod \"console-599f4f87f4-25xgn\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.530617 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.530565 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c2c98da9-5848-4b54-8034-d60ef7e81fcd-crio-socket\") pod \"insights-runtime-extractor-2sr94\" (UID: \"c2c98da9-5848-4b54-8034-d60ef7e81fcd\") " pod="openshift-insights/insights-runtime-extractor-2sr94" Jul 26 16:09:21.530795 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.530628 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2c98da9-5848-4b54-8034-d60ef7e81fcd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2sr94\" (UID: \"c2c98da9-5848-4b54-8034-d60ef7e81fcd\") " pod="openshift-insights/insights-runtime-extractor-2sr94" Jul 26 16:09:21.530795 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.530662 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55p6d\" (UniqueName: \"kubernetes.io/projected/62ca1095-ccae-444f-b8e6-2625663e191f-kube-api-access-55p6d\") pod \"ingress-canary-hsc48\" (UID: \"62ca1095-ccae-444f-b8e6-2625663e191f\") " pod="openshift-ingress-canary/ingress-canary-hsc48" Jul 26 16:09:21.530795 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.530684 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/358697ed-6e24-42e2-9687-eee030fe181c-service-ca\") pod \"console-599f4f87f4-25xgn\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.530795 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.530719 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62ca1095-ccae-444f-b8e6-2625663e191f-cert\") pod \"ingress-canary-hsc48\" (UID: \"62ca1095-ccae-444f-b8e6-2625663e191f\") " pod="openshift-ingress-canary/ingress-canary-hsc48" Jul 26 16:09:21.530795 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.530748 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkqlh\" (UniqueName: \"kubernetes.io/projected/c2c98da9-5848-4b54-8034-d60ef7e81fcd-kube-api-access-jkqlh\") pod \"insights-runtime-extractor-2sr94\" (UID: \"c2c98da9-5848-4b54-8034-d60ef7e81fcd\") " pod="openshift-insights/insights-runtime-extractor-2sr94" Jul 26 16:09:21.530795 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.530772 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cee3f58-a577-48df-9b42-ea4fba7ed3cf-tmp-dir\") pod \"dns-default-b27r6\" (UID: \"3cee3f58-a577-48df-9b42-ea4fba7ed3cf\") " pod="openshift-dns/dns-default-b27r6" Jul 26 16:09:21.531082 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.530799 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/358697ed-6e24-42e2-9687-eee030fe181c-console-oauth-config\") pod \"console-599f4f87f4-25xgn\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.531082 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.530853 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c2c98da9-5848-4b54-8034-d60ef7e81fcd-crio-socket\") pod \"insights-runtime-extractor-2sr94\" (UID: \"c2c98da9-5848-4b54-8034-d60ef7e81fcd\") " pod="openshift-insights/insights-runtime-extractor-2sr94" Jul 26 16:09:21.531082 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.530872 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c2c98da9-5848-4b54-8034-d60ef7e81fcd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2sr94\" (UID: \"c2c98da9-5848-4b54-8034-d60ef7e81fcd\") " pod="openshift-insights/insights-runtime-extractor-2sr94" Jul 26 16:09:21.531082 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.530915 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3cee3f58-a577-48df-9b42-ea4fba7ed3cf-config-volume\") pod \"dns-default-b27r6\" (UID: \"3cee3f58-a577-48df-9b42-ea4fba7ed3cf\") " pod="openshift-dns/dns-default-b27r6" Jul 26 16:09:21.531082 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.530945 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwjnn\" (UniqueName: \"kubernetes.io/projected/358697ed-6e24-42e2-9687-eee030fe181c-kube-api-access-hwjnn\") pod \"console-599f4f87f4-25xgn\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.531082 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.530984 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cee3f58-a577-48df-9b42-ea4fba7ed3cf-metrics-tls\") pod \"dns-default-b27r6\" (UID: \"3cee3f58-a577-48df-9b42-ea4fba7ed3cf\") " pod="openshift-dns/dns-default-b27r6" Jul 26 16:09:21.531082 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.531013 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/358697ed-6e24-42e2-9687-eee030fe181c-console-config\") pod \"console-599f4f87f4-25xgn\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.531082 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.531064 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/358697ed-6e24-42e2-9687-eee030fe181c-console-serving-cert\") pod \"console-599f4f87f4-25xgn\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.531486 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.531095 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/358697ed-6e24-42e2-9687-eee030fe181c-oauth-serving-cert\") pod \"console-599f4f87f4-25xgn\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.531486 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.531122 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2c98da9-5848-4b54-8034-d60ef7e81fcd-data-volume\") pod \"insights-runtime-extractor-2sr94\" (UID: \"c2c98da9-5848-4b54-8034-d60ef7e81fcd\") " pod="openshift-insights/insights-runtime-extractor-2sr94" Jul 26 16:09:21.531486 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.531145 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfldh\" (UniqueName: \"kubernetes.io/projected/3cee3f58-a577-48df-9b42-ea4fba7ed3cf-kube-api-access-wfldh\") pod \"dns-default-b27r6\" (UID: \"3cee3f58-a577-48df-9b42-ea4fba7ed3cf\") " pod="openshift-dns/dns-default-b27r6" Jul 26 16:09:21.531486 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.531389 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cee3f58-a577-48df-9b42-ea4fba7ed3cf-tmp-dir\") pod \"dns-default-b27r6\" (UID: \"3cee3f58-a577-48df-9b42-ea4fba7ed3cf\") " pod="openshift-dns/dns-default-b27r6" Jul 26 16:09:21.532602 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.531771 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3cee3f58-a577-48df-9b42-ea4fba7ed3cf-config-volume\") pod \"dns-default-b27r6\" (UID: \"3cee3f58-a577-48df-9b42-ea4fba7ed3cf\") " pod="openshift-dns/dns-default-b27r6" Jul 26 16:09:21.532602 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.531816 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/358697ed-6e24-42e2-9687-eee030fe181c-service-ca\") pod \"console-599f4f87f4-25xgn\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.532602 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.532015 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2c98da9-5848-4b54-8034-d60ef7e81fcd-data-volume\") pod \"insights-runtime-extractor-2sr94\" (UID: \"c2c98da9-5848-4b54-8034-d60ef7e81fcd\") " pod="openshift-insights/insights-runtime-extractor-2sr94" Jul 26 16:09:21.532602 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.532030 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/358697ed-6e24-42e2-9687-eee030fe181c-oauth-serving-cert\") pod \"console-599f4f87f4-25xgn\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.532602 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.532202 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c2c98da9-5848-4b54-8034-d60ef7e81fcd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2sr94\" (UID: \"c2c98da9-5848-4b54-8034-d60ef7e81fcd\") " pod="openshift-insights/insights-runtime-extractor-2sr94" Jul 26 16:09:21.532602 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.532559 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/358697ed-6e24-42e2-9687-eee030fe181c-console-config\") pod \"console-599f4f87f4-25xgn\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.535585 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.535560 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2c98da9-5848-4b54-8034-d60ef7e81fcd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2sr94\" (UID: \"c2c98da9-5848-4b54-8034-d60ef7e81fcd\") " pod="openshift-insights/insights-runtime-extractor-2sr94" Jul 26 16:09:21.535678 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.535609 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/358697ed-6e24-42e2-9687-eee030fe181c-console-oauth-config\") pod \"console-599f4f87f4-25xgn\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.535678 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.535634 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62ca1095-ccae-444f-b8e6-2625663e191f-cert\") pod \"ingress-canary-hsc48\" (UID: \"62ca1095-ccae-444f-b8e6-2625663e191f\") " pod="openshift-ingress-canary/ingress-canary-hsc48" Jul 26 16:09:21.535678 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.535616 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/358697ed-6e24-42e2-9687-eee030fe181c-console-serving-cert\") pod \"console-599f4f87f4-25xgn\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.536191 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.536164 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cee3f58-a577-48df-9b42-ea4fba7ed3cf-metrics-tls\") pod \"dns-default-b27r6\" (UID: \"3cee3f58-a577-48df-9b42-ea4fba7ed3cf\") " pod="openshift-dns/dns-default-b27r6" Jul 26 16:09:21.539258 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.539189 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkqlh\" (UniqueName: \"kubernetes.io/projected/c2c98da9-5848-4b54-8034-d60ef7e81fcd-kube-api-access-jkqlh\") pod \"insights-runtime-extractor-2sr94\" (UID: \"c2c98da9-5848-4b54-8034-d60ef7e81fcd\") " pod="openshift-insights/insights-runtime-extractor-2sr94" Jul 26 16:09:21.540090 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.540070 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwjnn\" (UniqueName: \"kubernetes.io/projected/358697ed-6e24-42e2-9687-eee030fe181c-kube-api-access-hwjnn\") pod \"console-599f4f87f4-25xgn\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.540192 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.540062 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55p6d\" (UniqueName: \"kubernetes.io/projected/62ca1095-ccae-444f-b8e6-2625663e191f-kube-api-access-55p6d\") pod \"ingress-canary-hsc48\" (UID: \"62ca1095-ccae-444f-b8e6-2625663e191f\") " pod="openshift-ingress-canary/ingress-canary-hsc48" Jul 26 16:09:21.540311 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.540296 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfldh\" (UniqueName: \"kubernetes.io/projected/3cee3f58-a577-48df-9b42-ea4fba7ed3cf-kube-api-access-wfldh\") pod \"dns-default-b27r6\" (UID: \"3cee3f58-a577-48df-9b42-ea4fba7ed3cf\") " pod="openshift-dns/dns-default-b27r6" Jul 26 16:09:21.615850 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.615815 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b27r6" Jul 26 16:09:21.623834 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.623811 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hsc48" Jul 26 16:09:21.632764 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.632745 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:21.643204 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.643184 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2sr94" Jul 26 16:09:21.884645 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.884599 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:09:21.884822 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.884655 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:09:21.888094 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.888070 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 26 16:09:21.888209 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.888091 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 26 16:09:21.888209 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.888121 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 26 16:09:21.888209 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:21.888070 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6nt9n\"" Jul 26 16:09:22.706424 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.706085 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74cf85ccf5-gzmp8"] Jul 26 16:09:22.711772 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.711645 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.731581 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.731547 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74cf85ccf5-gzmp8"] Jul 26 16:09:22.737292 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.737107 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 26 16:09:22.776450 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.776422 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-599f4f87f4-25xgn"] Jul 26 16:09:22.782394 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.782247 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2sr94"] Jul 26 16:09:22.782798 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:09:22.782776 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358697ed_6e24_42e2_9687_eee030fe181c.slice/crio-84e19cf467504f2cc2161720fc207eeadc569a5bc23acd0fd9a218e046c7eee9 WatchSource:0}: Error finding container 84e19cf467504f2cc2161720fc207eeadc569a5bc23acd0fd9a218e046c7eee9: Status 404 returned error can't find the container with id 84e19cf467504f2cc2161720fc207eeadc569a5bc23acd0fd9a218e046c7eee9 Jul 26 16:09:22.786203 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:09:22.786141 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c98da9_5848_4b54_8034_d60ef7e81fcd.slice/crio-0286ad1b2bea405e1fbaf59de16b2308b5f78e8175f2bd3312cede8e70dfc971 WatchSource:0}: Error finding container 0286ad1b2bea405e1fbaf59de16b2308b5f78e8175f2bd3312cede8e70dfc971: Status 404 returned error can't find the container with id 0286ad1b2bea405e1fbaf59de16b2308b5f78e8175f2bd3312cede8e70dfc971 Jul 26 16:09:22.786203 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.786144 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hsc48"] Jul 26 16:09:22.788829 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.788805 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b27r6"] Jul 26 16:09:22.790428 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:09:22.790401 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ca1095_ccae_444f_b8e6_2625663e191f.slice/crio-17e436c0e003da5d7076bad8caf68a3ab386b0a037027c05185ffb2dc780d513 WatchSource:0}: Error finding container 17e436c0e003da5d7076bad8caf68a3ab386b0a037027c05185ffb2dc780d513: Status 404 returned error can't find the container with id 17e436c0e003da5d7076bad8caf68a3ab386b0a037027c05185ffb2dc780d513 Jul 26 16:09:22.792276 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:09:22.792246 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cee3f58_a577_48df_9b42_ea4fba7ed3cf.slice/crio-0436b73f476227723e17758ce2b08fcc0a31d91fa9f33997f59ae76bf6098616 WatchSource:0}: Error finding container 0436b73f476227723e17758ce2b08fcc0a31d91fa9f33997f59ae76bf6098616: Status 404 returned error can't find the container with id 0436b73f476227723e17758ce2b08fcc0a31d91fa9f33997f59ae76bf6098616 Jul 26 16:09:22.842110 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.842014 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-trusted-ca-bundle\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.842110 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.842051 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-console-oauth-config\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.842110 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.842072 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-service-ca\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.842327 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.842208 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-console-serving-cert\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.842327 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.842272 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbxz6\" (UniqueName: \"kubernetes.io/projected/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-kube-api-access-fbxz6\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.842327 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.842298 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-oauth-serving-cert\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.842417 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.842340 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-console-config\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.885348 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.885324 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:09:22.888209 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.888189 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 26 16:09:22.888327 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.888295 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h6wls\"" Jul 26 16:09:22.943172 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.943125 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-console-serving-cert\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.943336 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.943183 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbxz6\" (UniqueName: \"kubernetes.io/projected/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-kube-api-access-fbxz6\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.943336 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.943310 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-oauth-serving-cert\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.943425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.943376 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-console-config\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.943425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.943415 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-trusted-ca-bundle\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.943511 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.943475 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-console-oauth-config\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.943511 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.943504 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-service-ca\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.944074 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.944050 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-console-config\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.944201 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.944108 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-oauth-serving-cert\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.944281 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.944247 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-service-ca\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.944281 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.944271 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-trusted-ca-bundle\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.947757 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.947735 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-console-oauth-config\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.947822 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.947771 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-console-serving-cert\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:22.951100 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:22.951081 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbxz6\" (UniqueName: \"kubernetes.io/projected/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-kube-api-access-fbxz6\") pod \"console-74cf85ccf5-gzmp8\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:23.036148 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:23.036120 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:23.154461 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:23.154432 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74cf85ccf5-gzmp8"] Jul 26 16:09:23.157573 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:09:23.157546 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d6e71f_f56f_4fc4_92e0_fed8eef0f686.slice/crio-f90c2c26d6d0699803b0731338e9351101829b01a61619c59cc2e796e4838cb3 WatchSource:0}: Error finding container f90c2c26d6d0699803b0731338e9351101829b01a61619c59cc2e796e4838cb3: Status 404 returned error can't find the container with id f90c2c26d6d0699803b0731338e9351101829b01a61619c59cc2e796e4838cb3 Jul 26 16:09:23.218520 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:23.218428 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599f4f87f4-25xgn" event={"ID":"358697ed-6e24-42e2-9687-eee030fe181c","Type":"ContainerStarted","Data":"84e19cf467504f2cc2161720fc207eeadc569a5bc23acd0fd9a218e046c7eee9"} Jul 26 16:09:23.220468 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:23.220428 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b27r6" event={"ID":"3cee3f58-a577-48df-9b42-ea4fba7ed3cf","Type":"ContainerStarted","Data":"0436b73f476227723e17758ce2b08fcc0a31d91fa9f33997f59ae76bf6098616"} Jul 26 16:09:23.228390 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:23.228357 2563 generic.go:356] "Generic (PLEG): container finished" podID="d83251cf-1af1-438a-9525-a79558f5f48f" containerID="4c0e7f842f0874001b334799f38acfd40280f1da8629f8c82746a3c3753a5a3e" exitCode=0 Jul 26 16:09:23.228514 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:23.228453 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bd5xf" event={"ID":"d83251cf-1af1-438a-9525-a79558f5f48f","Type":"ContainerDied","Data":"4c0e7f842f0874001b334799f38acfd40280f1da8629f8c82746a3c3753a5a3e"} Jul 26 16:09:23.242385 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:23.242270 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74cf85ccf5-gzmp8" event={"ID":"32d6e71f-f56f-4fc4-92e0-fed8eef0f686","Type":"ContainerStarted","Data":"f90c2c26d6d0699803b0731338e9351101829b01a61619c59cc2e796e4838cb3"} Jul 26 16:09:23.253501 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:23.253401 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hsc48" event={"ID":"62ca1095-ccae-444f-b8e6-2625663e191f","Type":"ContainerStarted","Data":"17e436c0e003da5d7076bad8caf68a3ab386b0a037027c05185ffb2dc780d513"} Jul 26 16:09:23.256397 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:23.256333 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2sr94" event={"ID":"c2c98da9-5848-4b54-8034-d60ef7e81fcd","Type":"ContainerStarted","Data":"0286ad1b2bea405e1fbaf59de16b2308b5f78e8175f2bd3312cede8e70dfc971"} Jul 26 16:09:24.286104 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:24.285295 2563 generic.go:356] "Generic (PLEG): container finished" podID="d83251cf-1af1-438a-9525-a79558f5f48f" containerID="4a8ed20b0d3798330d05a151a08015ffbc4bb030605cc42506fe7f4f1815499a" exitCode=0 Jul 26 16:09:24.286104 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:24.285369 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bd5xf" event={"ID":"d83251cf-1af1-438a-9525-a79558f5f48f","Type":"ContainerDied","Data":"4a8ed20b0d3798330d05a151a08015ffbc4bb030605cc42506fe7f4f1815499a"} Jul 26 16:09:25.306316 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:25.305820 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bd5xf" event={"ID":"d83251cf-1af1-438a-9525-a79558f5f48f","Type":"ContainerStarted","Data":"f651fa7cba60121090c0c9b32353585a2fd29c2ad7e81907d18af6274f836a30"} Jul 26 16:09:25.333325 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:25.330693 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bd5xf" podStartSLOduration=8.423871857 podStartE2EDuration="38.330674537s" totalImagesPullingTime="29.90680268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 16:08:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:25.329105908 +0000 UTC m=+38.910192481" watchObservedRunningTime="2026-07-26 16:09:25.330674537 +0000 UTC m=+38.911761091" Jul 26 16:09:32.326619 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:32.326513 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b27r6" event={"ID":"3cee3f58-a577-48df-9b42-ea4fba7ed3cf","Type":"ContainerStarted","Data":"8bb7183b3e38a00a4d881d8a9c622c5b99110c6c69a29956c2f79949bbd934b3"} Jul 26 16:09:32.326619 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:32.326614 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b27r6" event={"ID":"3cee3f58-a577-48df-9b42-ea4fba7ed3cf","Type":"ContainerStarted","Data":"301d1cdf93e955a31d7bdcacfb589925141e2370e1215b0cc7e7bb7815bec044"} Jul 26 16:09:32.327131 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:32.326644 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b27r6" Jul 26 16:09:32.328035 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:32.327994 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74cf85ccf5-gzmp8" event={"ID":"32d6e71f-f56f-4fc4-92e0-fed8eef0f686","Type":"ContainerStarted","Data":"937ab77d7127a5962e8eacf69cb78b38bd69d2aa054e83fd6f690bbd933f7abc"} Jul 26 16:09:32.329366 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:32.329344 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hsc48" event={"ID":"62ca1095-ccae-444f-b8e6-2625663e191f","Type":"ContainerStarted","Data":"90d247cd6fd9a14b4218c155c5054127cfeb208e702bd14c87747acb4c5ec113"} Jul 26 16:09:32.330783 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:32.330756 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2sr94" event={"ID":"c2c98da9-5848-4b54-8034-d60ef7e81fcd","Type":"ContainerStarted","Data":"a28a6da68a814d0bcb79a7c5544d82cb6f360d0efef4a98dd6c243f3f06d6490"} Jul 26 16:09:32.332164 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:32.332139 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599f4f87f4-25xgn" event={"ID":"358697ed-6e24-42e2-9687-eee030fe181c","Type":"ContainerStarted","Data":"d9a5e0fbb3da2aa3b003e18ed9fa0998564333843dc66a1c297c49524dd3a27c"} Jul 26 16:09:32.343643 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:32.343598 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-b27r6" podStartSLOduration=2.3300869 podStartE2EDuration="11.343586521s" totalImagesPullingTime="9.013499621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:32.343054488 +0000 UTC m=+45.924141059" watchObservedRunningTime="2026-07-26 16:09:32.343586521 +0000 UTC m=+45.924673137" Jul 26 16:09:32.360041 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:32.359989 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74cf85ccf5-gzmp8" podStartSLOduration=1.71137385 podStartE2EDuration="10.35997283s" totalImagesPullingTime="8.64859898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:32.359291186 +0000 UTC m=+45.940377756" watchObservedRunningTime="2026-07-26 16:09:32.35997283 +0000 UTC m=+45.941059401" Jul 26 16:09:32.377860 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:32.377817 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-599f4f87f4-25xgn" podStartSLOduration=2.354994135 podStartE2EDuration="11.377803178s" totalImagesPullingTime="9.022809043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:32.377302694 +0000 UTC m=+45.958389261" watchObservedRunningTime="2026-07-26 16:09:32.377803178 +0000 UTC m=+45.958889805" Jul 26 16:09:32.392473 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:32.392435 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hsc48" podStartSLOduration=2.376581026 podStartE2EDuration="11.392421339s" totalImagesPullingTime="9.015840313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:32.391722839 +0000 UTC m=+45.972809423" watchObservedRunningTime="2026-07-26 16:09:32.392421339 +0000 UTC m=+45.973507908" Jul 26 16:09:33.036678 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:33.036641 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:33.036852 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:33.036704 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:33.038337 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:33.038307 2563 patch_prober.go:28] interesting pod/console-74cf85ccf5-gzmp8 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" start-of-body= Jul 26 16:09:33.038463 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:33.038361 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-74cf85ccf5-gzmp8" podUID="32d6e71f-f56f-4fc4-92e0-fed8eef0f686" containerName="console" probeResult="failure" output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" Jul 26 16:09:37.351637 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:37.351598 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2sr94" event={"ID":"c2c98da9-5848-4b54-8034-d60ef7e81fcd","Type":"ContainerStarted","Data":"05846676b902fc70b67a586a091cb22fbd422e184ef8e2f90c6895289f5e7f80"} Jul 26 16:09:39.086042 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.086010 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf"] Jul 26 16:09:39.125255 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.125165 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf"] Jul 26 16:09:39.127997 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.125526 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh"] Jul 26 16:09:39.127997 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.126852 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" Jul 26 16:09:39.131207 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.130534 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 26 16:09:39.131207 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.130757 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5dh5d\"" Jul 26 16:09:39.131207 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.130797 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 26 16:09:39.131207 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.130852 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 26 16:09:39.131207 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.130989 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 26 16:09:39.131207 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.131100 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 26 16:09:39.175456 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.175420 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ffc8cc6-21c2-4827-959f-25d6cfefffa2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-4h5nf\" (UID: \"6ffc8cc6-21c2-4827-959f-25d6cfefffa2\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" Jul 26 16:09:39.175639 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.175473 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ffc8cc6-21c2-4827-959f-25d6cfefffa2-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-4h5nf\" (UID: \"6ffc8cc6-21c2-4827-959f-25d6cfefffa2\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" Jul 26 16:09:39.175639 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.175506 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmqvr\" (UniqueName: \"kubernetes.io/projected/6ffc8cc6-21c2-4827-959f-25d6cfefffa2-kube-api-access-hmqvr\") pod \"openshift-state-metrics-b8f49f5c6-4h5nf\" (UID: \"6ffc8cc6-21c2-4827-959f-25d6cfefffa2\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" Jul 26 16:09:39.175639 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.175622 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ffc8cc6-21c2-4827-959f-25d6cfefffa2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-4h5nf\" (UID: \"6ffc8cc6-21c2-4827-959f-25d6cfefffa2\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" Jul 26 16:09:39.178981 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.178952 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dcds8"] Jul 26 16:09:39.179120 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.179079 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.182301 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.182279 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 26 16:09:39.182418 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.182390 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 26 16:09:39.182544 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.182524 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-46mdp\"" Jul 26 16:09:39.182631 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.182560 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 26 16:09:39.214162 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.214121 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh"] Jul 26 16:09:39.214342 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.214211 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.217421 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.217400 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 26 16:09:39.217590 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.217464 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 26 16:09:39.217590 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.217404 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wrthp\"" Jul 26 16:09:39.217590 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.217519 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 26 16:09:39.276484 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.276447 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15339a8c-492a-4a85-9988-59d8942ae27d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.276666 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.276537 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/15339a8c-492a-4a85-9988-59d8942ae27d-root\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.276666 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.276568 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15339a8c-492a-4a85-9988-59d8942ae27d-metrics-client-ca\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.276666 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.276605 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ffc8cc6-21c2-4827-959f-25d6cfefffa2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-4h5nf\" (UID: \"6ffc8cc6-21c2-4827-959f-25d6cfefffa2\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" Jul 26 16:09:39.276666 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.276633 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15339a8c-492a-4a85-9988-59d8942ae27d-sys\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.276666 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.276659 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/15339a8c-492a-4a85-9988-59d8942ae27d-node-exporter-wtmp\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.276916 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.276689 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ffc8cc6-21c2-4827-959f-25d6cfefffa2-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-4h5nf\" (UID: \"6ffc8cc6-21c2-4827-959f-25d6cfefffa2\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" Jul 26 16:09:39.276916 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.276713 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4cd9bc19-660a-4d27-9af5-b34498225894-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.276916 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.276770 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmqvr\" (UniqueName: \"kubernetes.io/projected/6ffc8cc6-21c2-4827-959f-25d6cfefffa2-kube-api-access-hmqvr\") pod \"openshift-state-metrics-b8f49f5c6-4h5nf\" (UID: \"6ffc8cc6-21c2-4827-959f-25d6cfefffa2\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" Jul 26 16:09:39.276916 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.276802 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/15339a8c-492a-4a85-9988-59d8942ae27d-node-exporter-textfile\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.276916 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.276842 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/15339a8c-492a-4a85-9988-59d8942ae27d-node-exporter-tls\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.276916 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.276893 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs5q4\" (UniqueName: \"kubernetes.io/projected/15339a8c-492a-4a85-9988-59d8942ae27d-kube-api-access-qs5q4\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.277239 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.276972 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4cd9bc19-660a-4d27-9af5-b34498225894-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.277239 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.277002 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4cd9bc19-660a-4d27-9af5-b34498225894-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.277239 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.277043 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ffc8cc6-21c2-4827-959f-25d6cfefffa2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-4h5nf\" (UID: \"6ffc8cc6-21c2-4827-959f-25d6cfefffa2\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" Jul 26 16:09:39.277239 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.277077 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhjf5\" (UniqueName: \"kubernetes.io/projected/4cd9bc19-660a-4d27-9af5-b34498225894-kube-api-access-hhjf5\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.277239 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.277117 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cd9bc19-660a-4d27-9af5-b34498225894-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.277239 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.277144 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/15339a8c-492a-4a85-9988-59d8942ae27d-node-exporter-accelerators-collector-config\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.277239 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:39.277159 2563 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 26 16:09:39.277239 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.277208 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4cd9bc19-660a-4d27-9af5-b34498225894-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.277239 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:39.277234 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ffc8cc6-21c2-4827-959f-25d6cfefffa2-openshift-state-metrics-tls podName:6ffc8cc6-21c2-4827-959f-25d6cfefffa2 nodeName:}" failed. No retries permitted until 2026-07-26 16:09:39.77719876 +0000 UTC m=+53.358285310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6ffc8cc6-21c2-4827-959f-25d6cfefffa2-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-4h5nf" (UID: "6ffc8cc6-21c2-4827-959f-25d6cfefffa2") : secret "openshift-state-metrics-tls" not found Jul 26 16:09:39.277656 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.277364 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ffc8cc6-21c2-4827-959f-25d6cfefffa2-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-4h5nf\" (UID: \"6ffc8cc6-21c2-4827-959f-25d6cfefffa2\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" Jul 26 16:09:39.280202 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.280177 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ffc8cc6-21c2-4827-959f-25d6cfefffa2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-4h5nf\" (UID: \"6ffc8cc6-21c2-4827-959f-25d6cfefffa2\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" Jul 26 16:09:39.287957 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.287933 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmqvr\" (UniqueName: \"kubernetes.io/projected/6ffc8cc6-21c2-4827-959f-25d6cfefffa2-kube-api-access-hmqvr\") pod \"openshift-state-metrics-b8f49f5c6-4h5nf\" (UID: \"6ffc8cc6-21c2-4827-959f-25d6cfefffa2\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" Jul 26 16:09:39.378620 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.378543 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cd9bc19-660a-4d27-9af5-b34498225894-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.378620 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.378582 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/15339a8c-492a-4a85-9988-59d8942ae27d-node-exporter-accelerators-collector-config\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.378620 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.378606 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4cd9bc19-660a-4d27-9af5-b34498225894-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.378893 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.378653 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15339a8c-492a-4a85-9988-59d8942ae27d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.378893 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.378686 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/15339a8c-492a-4a85-9988-59d8942ae27d-root\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.378893 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.378709 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15339a8c-492a-4a85-9988-59d8942ae27d-metrics-client-ca\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.378893 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:39.378721 2563 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 26 16:09:39.378893 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.378738 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15339a8c-492a-4a85-9988-59d8942ae27d-sys\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.378893 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.378755 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/15339a8c-492a-4a85-9988-59d8942ae27d-node-exporter-wtmp\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.378893 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.378779 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4cd9bc19-660a-4d27-9af5-b34498225894-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.378893 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:39.378798 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd9bc19-660a-4d27-9af5-b34498225894-kube-state-metrics-tls podName:4cd9bc19-660a-4d27-9af5-b34498225894 nodeName:}" failed. No retries permitted until 2026-07-26 16:09:39.878775476 +0000 UTC m=+53.459862040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/4cd9bc19-660a-4d27-9af5-b34498225894-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-cjzbh" (UID: "4cd9bc19-660a-4d27-9af5-b34498225894") : secret "kube-state-metrics-tls" not found Jul 26 16:09:39.378893 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.378850 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/15339a8c-492a-4a85-9988-59d8942ae27d-node-exporter-textfile\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.378893 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.378885 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/15339a8c-492a-4a85-9988-59d8942ae27d-node-exporter-tls\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.379426 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.378924 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs5q4\" (UniqueName: \"kubernetes.io/projected/15339a8c-492a-4a85-9988-59d8942ae27d-kube-api-access-qs5q4\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.379426 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.378970 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4cd9bc19-660a-4d27-9af5-b34498225894-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.379426 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.378997 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4cd9bc19-660a-4d27-9af5-b34498225894-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.379426 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.379047 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhjf5\" (UniqueName: \"kubernetes.io/projected/4cd9bc19-660a-4d27-9af5-b34498225894-kube-api-access-hhjf5\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.379426 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.379303 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/15339a8c-492a-4a85-9988-59d8942ae27d-node-exporter-accelerators-collector-config\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.379426 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.379367 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4cd9bc19-660a-4d27-9af5-b34498225894-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.379426 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.379381 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15339a8c-492a-4a85-9988-59d8942ae27d-sys\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.379728 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.379550 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15339a8c-492a-4a85-9988-59d8942ae27d-metrics-client-ca\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.379728 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.379596 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/15339a8c-492a-4a85-9988-59d8942ae27d-node-exporter-wtmp\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.379728 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.379608 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/15339a8c-492a-4a85-9988-59d8942ae27d-root\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.379728 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.379654 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/15339a8c-492a-4a85-9988-59d8942ae27d-node-exporter-textfile\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.379938 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.379838 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4cd9bc19-660a-4d27-9af5-b34498225894-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.380190 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.380163 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4cd9bc19-660a-4d27-9af5-b34498225894-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.381520 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.381499 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4cd9bc19-660a-4d27-9af5-b34498225894-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.381983 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.381965 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/15339a8c-492a-4a85-9988-59d8942ae27d-node-exporter-tls\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.382070 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.382054 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15339a8c-492a-4a85-9988-59d8942ae27d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.389953 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.389932 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhjf5\" (UniqueName: \"kubernetes.io/projected/4cd9bc19-660a-4d27-9af5-b34498225894-kube-api-access-hhjf5\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.390142 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.390119 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs5q4\" (UniqueName: \"kubernetes.io/projected/15339a8c-492a-4a85-9988-59d8942ae27d-kube-api-access-qs5q4\") pod \"node-exporter-dcds8\" (UID: \"15339a8c-492a-4a85-9988-59d8942ae27d\") " pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.525550 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.525514 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dcds8" Jul 26 16:09:39.781330 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.781299 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ffc8cc6-21c2-4827-959f-25d6cfefffa2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-4h5nf\" (UID: \"6ffc8cc6-21c2-4827-959f-25d6cfefffa2\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" Jul 26 16:09:39.781524 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:39.781480 2563 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 26 16:09:39.781595 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:39.781574 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ffc8cc6-21c2-4827-959f-25d6cfefffa2-openshift-state-metrics-tls podName:6ffc8cc6-21c2-4827-959f-25d6cfefffa2 nodeName:}" failed. No retries permitted until 2026-07-26 16:09:40.781551616 +0000 UTC m=+54.362638184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6ffc8cc6-21c2-4827-959f-25d6cfefffa2-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-4h5nf" (UID: "6ffc8cc6-21c2-4827-959f-25d6cfefffa2") : secret "openshift-state-metrics-tls" not found Jul 26 16:09:39.882476 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:39.882432 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cd9bc19-660a-4d27-9af5-b34498225894-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:39.882664 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:39.882609 2563 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 26 16:09:39.882730 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:39.882692 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd9bc19-660a-4d27-9af5-b34498225894-kube-state-metrics-tls podName:4cd9bc19-660a-4d27-9af5-b34498225894 nodeName:}" failed. No retries permitted until 2026-07-26 16:09:40.882671588 +0000 UTC m=+54.463758147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/4cd9bc19-660a-4d27-9af5-b34498225894-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-cjzbh" (UID: "4cd9bc19-660a-4d27-9af5-b34498225894") : secret "kube-state-metrics-tls" not found Jul 26 16:09:40.168503 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.168424 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 16:09:40.307578 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.307542 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 16:09:40.307718 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.307672 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.310614 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.310594 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 26 16:09:40.310750 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.310594 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 26 16:09:40.310750 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.310716 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 26 16:09:40.310970 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.310952 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 26 16:09:40.311043 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.310992 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 26 16:09:40.311043 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.310957 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 26 16:09:40.311140 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.310998 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 26 16:09:40.311140 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.311126 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 26 16:09:40.311394 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.311377 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xbhcw\"" Jul 26 16:09:40.311485 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.311450 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 26 16:09:40.386954 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.386923 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.387138 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.386963 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.387138 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.386996 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.387138 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.387037 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-web-config\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.387138 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.387063 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.387138 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.387102 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.387331 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.387144 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.387331 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.387190 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.387331 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.387261 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.387331 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.387286 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.387464 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.387333 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-config-out\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.387464 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.387370 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.387464 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.387399 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq684\" (UniqueName: \"kubernetes.io/projected/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-kube-api-access-zq684\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.418627 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:09:40.418566 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15339a8c_492a_4a85_9988_59d8942ae27d.slice/crio-8f01a627bda0965ec9358c3620ea895706f0292a39833dd17dd755868d15469c WatchSource:0}: Error finding container 8f01a627bda0965ec9358c3620ea895706f0292a39833dd17dd755868d15469c: Status 404 returned error can't find the container with id 8f01a627bda0965ec9358c3620ea895706f0292a39833dd17dd755868d15469c Jul 26 16:09:40.488515 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.488471 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-config-out\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.488515 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.488515 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.488732 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.488554 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq684\" (UniqueName: \"kubernetes.io/projected/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-kube-api-access-zq684\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.488732 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.488605 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.488732 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.488630 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.488732 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.488656 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.488732 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.488694 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-web-config\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.488732 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.488723 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.489156 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.488748 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.489156 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.488815 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.489156 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.488879 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.489156 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.488912 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.489156 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.488939 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.489894 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.489688 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.489894 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.489779 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.490056 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.490023 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.491600 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.491574 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.491706 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.491584 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.491706 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.491653 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-config-out\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.492265 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.492240 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.492395 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.492375 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-web-config\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.492710 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.492683 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.492919 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.492897 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.493541 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.493523 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.493867 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.493852 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.496497 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.496477 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq684\" (UniqueName: \"kubernetes.io/projected/ec858743-e2e3-4428-8b5a-07e1c4f4a1b3-kube-api-access-zq684\") pod \"alertmanager-main-0\" (UID: \"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.617073 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.617042 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 26 16:09:40.775166 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.775133 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 16:09:40.777859 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:09:40.777835 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec858743_e2e3_4428_8b5a_07e1c4f4a1b3.slice/crio-c79060a167774b81ef19ba59d86d23da14d9b7285fa361fc045c7a3ccbaf72ce WatchSource:0}: Error finding container c79060a167774b81ef19ba59d86d23da14d9b7285fa361fc045c7a3ccbaf72ce: Status 404 returned error can't find the container with id c79060a167774b81ef19ba59d86d23da14d9b7285fa361fc045c7a3ccbaf72ce Jul 26 16:09:40.791955 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.791921 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ffc8cc6-21c2-4827-959f-25d6cfefffa2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-4h5nf\" (UID: \"6ffc8cc6-21c2-4827-959f-25d6cfefffa2\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" Jul 26 16:09:40.794299 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.794278 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ffc8cc6-21c2-4827-959f-25d6cfefffa2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-4h5nf\" (UID: \"6ffc8cc6-21c2-4827-959f-25d6cfefffa2\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" Jul 26 16:09:40.892653 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.892623 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cd9bc19-660a-4d27-9af5-b34498225894-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:40.895690 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.895661 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cd9bc19-660a-4d27-9af5-b34498225894-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-cjzbh\" (UID: \"4cd9bc19-660a-4d27-9af5-b34498225894\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:40.938914 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.938836 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" Jul 26 16:09:40.988974 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:40.988933 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" Jul 26 16:09:41.082825 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:41.082791 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf"] Jul 26 16:09:41.137076 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:41.137040 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh"] Jul 26 16:09:41.236068 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:09:41.235990 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ffc8cc6_21c2_4827_959f_25d6cfefffa2.slice/crio-28dd8f71c7716c0b59c552b2647863d94053b9badeef7e440a34a815a09c961f WatchSource:0}: Error finding container 28dd8f71c7716c0b59c552b2647863d94053b9badeef7e440a34a815a09c961f: Status 404 returned error can't find the container with id 28dd8f71c7716c0b59c552b2647863d94053b9badeef7e440a34a815a09c961f Jul 26 16:09:41.236517 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:09:41.236495 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd9bc19_660a_4d27_9af5_b34498225894.slice/crio-30311b74f915009958fc7d18eba87c44f089f2b7f0fd786dd7340b52697aa2dd WatchSource:0}: Error finding container 30311b74f915009958fc7d18eba87c44f089f2b7f0fd786dd7340b52697aa2dd: Status 404 returned error can't find the container with id 30311b74f915009958fc7d18eba87c44f089f2b7f0fd786dd7340b52697aa2dd Jul 26 16:09:41.367444 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:41.367408 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2sr94" event={"ID":"c2c98da9-5848-4b54-8034-d60ef7e81fcd","Type":"ContainerStarted","Data":"101512f5e5d77892d6e6662fe48ee95092b08fcca3a7c699b7674a1780da87fd"} Jul 26 16:09:41.369570 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:41.369541 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dcds8" event={"ID":"15339a8c-492a-4a85-9988-59d8942ae27d","Type":"ContainerStarted","Data":"87370bea2ba97f308253565d770071ba25d74e29fa20374c06081941578750c1"} Jul 26 16:09:41.369673 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:41.369598 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dcds8" event={"ID":"15339a8c-492a-4a85-9988-59d8942ae27d","Type":"ContainerStarted","Data":"8f01a627bda0965ec9358c3620ea895706f0292a39833dd17dd755868d15469c"} Jul 26 16:09:41.370778 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:41.370756 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" event={"ID":"4cd9bc19-660a-4d27-9af5-b34498225894","Type":"ContainerStarted","Data":"30311b74f915009958fc7d18eba87c44f089f2b7f0fd786dd7340b52697aa2dd"} Jul 26 16:09:41.372115 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:41.372089 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3","Type":"ContainerStarted","Data":"c79060a167774b81ef19ba59d86d23da14d9b7285fa361fc045c7a3ccbaf72ce"} Jul 26 16:09:41.373991 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:41.373972 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" event={"ID":"6ffc8cc6-21c2-4827-959f-25d6cfefffa2","Type":"ContainerStarted","Data":"6f48ee56cb857dc5bbda244b2b13392246d0a0d58057ec1e5793c67ac19174d2"} Jul 26 16:09:41.374069 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:41.374012 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" event={"ID":"6ffc8cc6-21c2-4827-959f-25d6cfefffa2","Type":"ContainerStarted","Data":"17b787d9e35002efb21a34b9d5b0ecd265222c040d78c327f0ffca667e6ceec1"} Jul 26 16:09:41.374069 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:41.374030 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" event={"ID":"6ffc8cc6-21c2-4827-959f-25d6cfefffa2","Type":"ContainerStarted","Data":"28dd8f71c7716c0b59c552b2647863d94053b9badeef7e440a34a815a09c961f"} Jul 26 16:09:41.384629 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:41.384585 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2sr94" podStartSLOduration=2.682298985 podStartE2EDuration="20.384571172s" totalImagesPullingTime="17.702272187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 16:09:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:41.384282433 +0000 UTC m=+54.965369005" watchObservedRunningTime="2026-07-26 16:09:41.384571172 +0000 UTC m=+54.965657741" Jul 26 16:09:41.633132 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:41.633092 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:41.633344 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:41.633165 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:41.638900 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:41.638878 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:42.338964 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:42.338932 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b27r6" Jul 26 16:09:42.381382 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:42.380617 2563 generic.go:356] "Generic (PLEG): container finished" podID="ec858743-e2e3-4428-8b5a-07e1c4f4a1b3" containerID="5aa39119e5c758125ace6aca02b68b400a7b38ea1dedc140455b1a81e5d1a6c0" exitCode=0 Jul 26 16:09:42.381382 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:42.380775 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3","Type":"ContainerDied","Data":"5aa39119e5c758125ace6aca02b68b400a7b38ea1dedc140455b1a81e5d1a6c0"} Jul 26 16:09:42.386727 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:42.386695 2563 generic.go:356] "Generic (PLEG): container finished" podID="15339a8c-492a-4a85-9988-59d8942ae27d" containerID="87370bea2ba97f308253565d770071ba25d74e29fa20374c06081941578750c1" exitCode=0 Jul 26 16:09:42.387078 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:42.386993 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dcds8" event={"ID":"15339a8c-492a-4a85-9988-59d8942ae27d","Type":"ContainerDied","Data":"87370bea2ba97f308253565d770071ba25d74e29fa20374c06081941578750c1"} Jul 26 16:09:42.391675 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:42.391653 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:09:43.043279 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.043202 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:43.049252 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.049183 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:09:43.108974 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.108898 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-599f4f87f4-25xgn"] Jul 26 16:09:43.393517 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.393479 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dcds8" event={"ID":"15339a8c-492a-4a85-9988-59d8942ae27d","Type":"ContainerStarted","Data":"0e1675961e9ac7a9da4ea7901d716a0dec2996f93dfb846ba6e62dcbd50f5157"} Jul 26 16:09:43.393964 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.393551 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dcds8" event={"ID":"15339a8c-492a-4a85-9988-59d8942ae27d","Type":"ContainerStarted","Data":"5609a68fa47c1d802f0badb7a30041a226051a33d2a7d0417563c0077ecf5cb6"} Jul 26 16:09:43.395760 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.395725 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" event={"ID":"4cd9bc19-660a-4d27-9af5-b34498225894","Type":"ContainerStarted","Data":"f55c174ddcb39bfbca392b347b4d816d764f68a69de450c4f3c542ba2c107496"} Jul 26 16:09:43.395871 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.395786 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" event={"ID":"4cd9bc19-660a-4d27-9af5-b34498225894","Type":"ContainerStarted","Data":"1185bfda25319ef6121460573430ac7494fabcfc3fa744ac5587dd08fe8b9fcb"} Jul 26 16:09:43.395871 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.395812 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" event={"ID":"4cd9bc19-660a-4d27-9af5-b34498225894","Type":"ContainerStarted","Data":"73c71d1dd438a09e397073b9b5e8fd17bfdde91ac837c0b25fb3fbb6fc7d36a8"} Jul 26 16:09:43.397869 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.397810 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" event={"ID":"6ffc8cc6-21c2-4827-959f-25d6cfefffa2","Type":"ContainerStarted","Data":"48e01de24bc43fea580d07841824331d60ab5bb125f11feb0007e65749757a90"} Jul 26 16:09:43.420812 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.420751 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dcds8" podStartSLOduration=3.5039964279999998 podStartE2EDuration="4.420732125s" totalImagesPullingTime="840.651514ms" totalInitContainerRuntime="76.084183ms" isStatefulPod=true podCreationTimestamp="2026-07-26 16:09:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:43.41985195 +0000 UTC m=+57.000938513" watchObservedRunningTime="2026-07-26 16:09:43.420732125 +0000 UTC m=+57.001818696" Jul 26 16:09:43.445323 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.445267 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cjzbh" podStartSLOduration=2.772491455 podStartE2EDuration="4.445249898s" totalImagesPullingTime="1.672758443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:09:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:43.444617447 +0000 UTC m=+57.025704017" watchObservedRunningTime="2026-07-26 16:09:43.445249898 +0000 UTC m=+57.026336446" Jul 26 16:09:43.461961 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.461905 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-4h5nf" podStartSLOduration=2.919884531 podStartE2EDuration="4.46188594s" totalImagesPullingTime="1.542001409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:09:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:43.46113491 +0000 UTC m=+57.042221484" watchObservedRunningTime="2026-07-26 16:09:43.46188594 +0000 UTC m=+57.042972513" Jul 26 16:09:43.880014 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.879980 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-jmztv"] Jul 26 16:09:43.884512 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.884492 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmztv" Jul 26 16:09:43.887204 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.887180 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 26 16:09:43.887345 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.887187 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-blzxq\"" Jul 26 16:09:43.894153 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.894126 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-jmztv"] Jul 26 16:09:43.920800 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.920774 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e9a61391-e4db-44b4-900b-4d544bc9f39f-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-jmztv\" (UID: \"e9a61391-e4db-44b4-900b-4d544bc9f39f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmztv" Jul 26 16:09:43.920957 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:43.920817 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m8tc\" (UniqueName: \"kubernetes.io/projected/e9a61391-e4db-44b4-900b-4d544bc9f39f-kube-api-access-4m8tc\") pod \"monitoring-plugin-68b7fc9574-jmztv\" (UID: \"e9a61391-e4db-44b4-900b-4d544bc9f39f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmztv" Jul 26 16:09:44.021663 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:44.021630 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e9a61391-e4db-44b4-900b-4d544bc9f39f-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-jmztv\" (UID: \"e9a61391-e4db-44b4-900b-4d544bc9f39f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmztv" Jul 26 16:09:44.021816 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:44.021683 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m8tc\" (UniqueName: \"kubernetes.io/projected/e9a61391-e4db-44b4-900b-4d544bc9f39f-kube-api-access-4m8tc\") pod \"monitoring-plugin-68b7fc9574-jmztv\" (UID: \"e9a61391-e4db-44b4-900b-4d544bc9f39f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmztv" Jul 26 16:09:44.021816 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:44.021774 2563 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 26 16:09:44.021905 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:09:44.021856 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9a61391-e4db-44b4-900b-4d544bc9f39f-monitoring-plugin-cert podName:e9a61391-e4db-44b4-900b-4d544bc9f39f nodeName:}" failed. No retries permitted until 2026-07-26 16:09:44.521835016 +0000 UTC m=+58.102921581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/e9a61391-e4db-44b4-900b-4d544bc9f39f-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-jmztv" (UID: "e9a61391-e4db-44b4-900b-4d544bc9f39f") : secret "monitoring-plugin-cert" not found Jul 26 16:09:44.030421 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:44.030383 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m8tc\" (UniqueName: \"kubernetes.io/projected/e9a61391-e4db-44b4-900b-4d544bc9f39f-kube-api-access-4m8tc\") pod \"monitoring-plugin-68b7fc9574-jmztv\" (UID: \"e9a61391-e4db-44b4-900b-4d544bc9f39f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmztv" Jul 26 16:09:44.407265 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:44.407140 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3","Type":"ContainerStarted","Data":"a4fe41dc2d1dab5792a14d4f6f1250442e24d159d920f7d3e27920513885ae84"} Jul 26 16:09:44.407265 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:44.407249 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3","Type":"ContainerStarted","Data":"347849c81c8f168f9d5a6a6dabcad5ab0c3327de75b5cf6d94cb9f4a0b9784d2"} Jul 26 16:09:44.407752 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:44.407291 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3","Type":"ContainerStarted","Data":"538c07043a47923c756b9e2f0badd98f822b594f758cec92ac7a026b3ffd8df9"} Jul 26 16:09:44.407752 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:44.407329 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3","Type":"ContainerStarted","Data":"caf43527344cd14a68e53a34edebf4194c475c4ceff2ac020b67fa6b33ad4486"} Jul 26 16:09:44.407752 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:44.407367 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3","Type":"ContainerStarted","Data":"943f95dece9bc456f7a820c23ee8b6818fc4739ca016f46288bd0bdc5d2813d8"} Jul 26 16:09:44.525551 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:44.525512 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e9a61391-e4db-44b4-900b-4d544bc9f39f-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-jmztv\" (UID: \"e9a61391-e4db-44b4-900b-4d544bc9f39f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmztv" Jul 26 16:09:44.527799 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:44.527777 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e9a61391-e4db-44b4-900b-4d544bc9f39f-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-jmztv\" (UID: \"e9a61391-e4db-44b4-900b-4d544bc9f39f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmztv" Jul 26 16:09:44.796610 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:44.796579 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmztv" Jul 26 16:09:45.037091 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.037063 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-jmztv"] Jul 26 16:09:45.041542 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:09:45.041518 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a61391_e4db_44b4_900b_4d544bc9f39f.slice/crio-7c0480e634b85ed5f2f0491d10ece36c5eef504d4853d8023534eab8d87ed87f WatchSource:0}: Error finding container 7c0480e634b85ed5f2f0491d10ece36c5eef504d4853d8023534eab8d87ed87f: Status 404 returned error can't find the container with id 7c0480e634b85ed5f2f0491d10ece36c5eef504d4853d8023534eab8d87ed87f Jul 26 16:09:45.411472 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.411428 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmztv" event={"ID":"e9a61391-e4db-44b4-900b-4d544bc9f39f","Type":"ContainerStarted","Data":"7c0480e634b85ed5f2f0491d10ece36c5eef504d4853d8023534eab8d87ed87f"} Jul 26 16:09:45.414977 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.414947 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ec858743-e2e3-4428-8b5a-07e1c4f4a1b3","Type":"ContainerStarted","Data":"044584db08b11c09cb5e35ed17fa3fa1fe1d7f5f386eb15f18f3e60a47a7e5ec"} Jul 26 16:09:45.456276 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.456201 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.93653695 podStartE2EDuration="5.456180924s" totalImagesPullingTime="3.501952963s" totalInitContainerRuntime="17.691011ms" isStatefulPod=false podCreationTimestamp="2026-07-26 16:09:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:45.454031983 +0000 UTC m=+59.035118566" watchObservedRunningTime="2026-07-26 16:09:45.456180924 +0000 UTC m=+59.037267496" Jul 26 16:09:45.540669 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.540633 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9df5c9885-6snv6"] Jul 26 16:09:45.543937 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.543921 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.554171 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.554145 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9df5c9885-6snv6"] Jul 26 16:09:45.635904 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.635867 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-console-config\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.636064 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.635914 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-console-serving-cert\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.636064 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.636005 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-oauth-serving-cert\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.636064 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.636043 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-console-oauth-config\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.636211 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.636065 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97jc5\" (UniqueName: \"kubernetes.io/projected/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-kube-api-access-97jc5\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.636211 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.636172 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-trusted-ca-bundle\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.636339 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.636295 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-service-ca\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.737694 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.737613 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-console-config\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.737694 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.737655 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-console-serving-cert\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.737907 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.737696 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-oauth-serving-cert\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.737907 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.737719 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-console-oauth-config\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.737907 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.737743 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97jc5\" (UniqueName: \"kubernetes.io/projected/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-kube-api-access-97jc5\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.738054 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.737908 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-trusted-ca-bundle\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.738054 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.738034 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-service-ca\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.738433 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.738409 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-console-config\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.738573 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.738525 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-oauth-serving-cert\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.738847 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.738824 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-service-ca\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.739172 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.739148 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-trusted-ca-bundle\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.740288 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.740267 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-console-oauth-config\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.740414 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.740398 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-console-serving-cert\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.746007 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.745989 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97jc5\" (UniqueName: \"kubernetes.io/projected/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-kube-api-access-97jc5\") pod \"console-9df5c9885-6snv6\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.853390 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.853350 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:45.995548 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:45.995445 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9df5c9885-6snv6"] Jul 26 16:09:45.998098 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:09:45.998065 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6063ef00_8ee2_4b3a_8baa_d7027e0f562b.slice/crio-c9aa7f22072cbfcb093ba5eccf13e62ea8284b0d18273338f7a2ab24d8d5c231 WatchSource:0}: Error finding container c9aa7f22072cbfcb093ba5eccf13e62ea8284b0d18273338f7a2ab24d8d5c231: Status 404 returned error can't find the container with id c9aa7f22072cbfcb093ba5eccf13e62ea8284b0d18273338f7a2ab24d8d5c231 Jul 26 16:09:46.423130 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:46.423092 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9df5c9885-6snv6" event={"ID":"6063ef00-8ee2-4b3a-8baa-d7027e0f562b","Type":"ContainerStarted","Data":"d43f340138e4b1df1d5b77061c57f25b2619e2d505e64d847f303068bda71a94"} Jul 26 16:09:46.423507 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:46.423149 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9df5c9885-6snv6" event={"ID":"6063ef00-8ee2-4b3a-8baa-d7027e0f562b","Type":"ContainerStarted","Data":"c9aa7f22072cbfcb093ba5eccf13e62ea8284b0d18273338f7a2ab24d8d5c231"} Jul 26 16:09:46.445770 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:46.443368 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9df5c9885-6snv6" podStartSLOduration=1.443347157 podStartE2EDuration="1.443347157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:46.441345058 +0000 UTC m=+60.022431640" watchObservedRunningTime="2026-07-26 16:09:46.443347157 +0000 UTC m=+60.024433731" Jul 26 16:09:47.210558 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:47.210479 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4zvz" Jul 26 16:09:47.427893 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:47.427851 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmztv" event={"ID":"e9a61391-e4db-44b4-900b-4d544bc9f39f","Type":"ContainerStarted","Data":"24680e83df546c579e35db9f941b6668593f87edc9abb4a887c92d0b5048f912"} Jul 26 16:09:47.444468 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:47.444422 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmztv" podStartSLOduration=2.625540399 podStartE2EDuration="4.444406119s" totalImagesPullingTime="1.81886572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:47.443570847 +0000 UTC m=+61.024657418" watchObservedRunningTime="2026-07-26 16:09:47.444406119 +0000 UTC m=+61.025492720" Jul 26 16:09:48.431825 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:48.431790 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmztv" Jul 26 16:09:48.436858 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:48.436834 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmztv" Jul 26 16:09:52.602809 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:52.602765 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs\") pod \"network-metrics-daemon-qd8ng\" (UID: \"bc7a4aef-1953-45ab-84db-8338395b0b25\") " pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:09:52.605558 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:52.605536 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 26 16:09:52.615905 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:52.615874 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7a4aef-1953-45ab-84db-8338395b0b25-metrics-certs\") pod \"network-metrics-daemon-qd8ng\" (UID: \"bc7a4aef-1953-45ab-84db-8338395b0b25\") " pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:09:52.703565 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:52.703532 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret\") pod \"global-pull-secret-syncer-hs2qb\" (UID: \"9f8db869-87b0-4a2c-b292-b78f22f3a391\") " pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:09:52.706528 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:52.706509 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 26 16:09:52.716723 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:52.716704 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f8db869-87b0-4a2c-b292-b78f22f3a391-original-pull-secret\") pod \"global-pull-secret-syncer-hs2qb\" (UID: \"9f8db869-87b0-4a2c-b292-b78f22f3a391\") " pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:09:52.796151 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:52.796117 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hs2qb" Jul 26 16:09:52.803932 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:52.803900 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjgxp\" (UniqueName: \"kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp\") pod \"network-check-target-dndsz\" (UID: \"1d66c1fe-74b0-44a3-b892-bd68dd3ffc51\") " pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:09:52.806850 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:52.806816 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 26 16:09:52.817535 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:52.817507 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 26 16:09:52.828209 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:52.828184 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjgxp\" (UniqueName: \"kubernetes.io/projected/1d66c1fe-74b0-44a3-b892-bd68dd3ffc51-kube-api-access-fjgxp\") pod \"network-check-target-dndsz\" (UID: \"1d66c1fe-74b0-44a3-b892-bd68dd3ffc51\") " pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:09:52.905415 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:52.905388 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h6wls\"" Jul 26 16:09:52.912601 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:52.912573 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8ng" Jul 26 16:09:52.926151 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:52.926125 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hs2qb"] Jul 26 16:09:52.929067 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:09:52.929043 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8db869_87b0_4a2c_b292_b78f22f3a391.slice/crio-9506af59a8a9d0356f56ea832011cab8030f532cf42d47c7da7806febf21f0f5 WatchSource:0}: Error finding container 9506af59a8a9d0356f56ea832011cab8030f532cf42d47c7da7806febf21f0f5: Status 404 returned error can't find the container with id 9506af59a8a9d0356f56ea832011cab8030f532cf42d47c7da7806febf21f0f5 Jul 26 16:09:53.041508 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:53.041469 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qd8ng"] Jul 26 16:09:53.044921 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:09:53.044891 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7a4aef_1953_45ab_84db_8338395b0b25.slice/crio-41e5bd29766139883d271e42792a974b9ef17e016d94a36a156e466fcfb61aae WatchSource:0}: Error finding container 41e5bd29766139883d271e42792a974b9ef17e016d94a36a156e466fcfb61aae: Status 404 returned error can't find the container with id 41e5bd29766139883d271e42792a974b9ef17e016d94a36a156e466fcfb61aae Jul 26 16:09:53.105051 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:53.105018 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6nt9n\"" Jul 26 16:09:53.113348 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:53.113325 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:09:53.251126 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:53.251091 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dndsz"] Jul 26 16:09:53.253526 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:09:53.253500 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d66c1fe_74b0_44a3_b892_bd68dd3ffc51.slice/crio-8e8cc77e512e33ea0bd621aaf88e5b5afd0a7340e6ed141e744822a3001f3a18 WatchSource:0}: Error finding container 8e8cc77e512e33ea0bd621aaf88e5b5afd0a7340e6ed141e744822a3001f3a18: Status 404 returned error can't find the container with id 8e8cc77e512e33ea0bd621aaf88e5b5afd0a7340e6ed141e744822a3001f3a18 Jul 26 16:09:53.451627 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:53.451538 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hs2qb" event={"ID":"9f8db869-87b0-4a2c-b292-b78f22f3a391","Type":"ContainerStarted","Data":"9506af59a8a9d0356f56ea832011cab8030f532cf42d47c7da7806febf21f0f5"} Jul 26 16:09:53.452883 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:53.452843 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dndsz" event={"ID":"1d66c1fe-74b0-44a3-b892-bd68dd3ffc51","Type":"ContainerStarted","Data":"8e8cc77e512e33ea0bd621aaf88e5b5afd0a7340e6ed141e744822a3001f3a18"} Jul 26 16:09:53.454044 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:53.454012 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qd8ng" event={"ID":"bc7a4aef-1953-45ab-84db-8338395b0b25","Type":"ContainerStarted","Data":"41e5bd29766139883d271e42792a974b9ef17e016d94a36a156e466fcfb61aae"} Jul 26 16:09:55.467187 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:55.467143 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qd8ng" event={"ID":"bc7a4aef-1953-45ab-84db-8338395b0b25","Type":"ContainerStarted","Data":"f2329ed19d894abaa7cc43fb3204ccb0575d98a6679638df96babd9f2c170bca"} Jul 26 16:09:55.467630 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:55.467236 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qd8ng" event={"ID":"bc7a4aef-1953-45ab-84db-8338395b0b25","Type":"ContainerStarted","Data":"7121ced92ae53a2dea36d146c911e625481492ebeae4311ba3f1f1168deff215"} Jul 26 16:09:55.484778 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:55.484724 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qd8ng" podStartSLOduration=66.969890501 podStartE2EDuration="1m8.484707848s" totalImagesPullingTime="1.514817347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:55.483888766 +0000 UTC m=+69.064975516" watchObservedRunningTime="2026-07-26 16:09:55.484707848 +0000 UTC m=+69.065794419" Jul 26 16:09:55.854278 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:55.854246 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:55.854479 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:55.854331 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:55.859769 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:55.859744 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:56.476900 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:56.476871 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:09:56.525669 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:56.525629 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74cf85ccf5-gzmp8"] Jul 26 16:09:58.482620 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:58.482579 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hs2qb" event={"ID":"9f8db869-87b0-4a2c-b292-b78f22f3a391","Type":"ContainerStarted","Data":"f2477148a96222dcb47de2acce2690ba8ad807156b3fbefe76bfce1cd77a2da6"} Jul 26 16:09:58.484025 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:58.483998 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dndsz" event={"ID":"1d66c1fe-74b0-44a3-b892-bd68dd3ffc51","Type":"ContainerStarted","Data":"c4904d1b1156519754f4d6e1d3d8beafeeeb3bec4ea6410c5d9d4c9297e628f0"} Jul 26 16:09:58.484121 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:58.484110 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:09:58.498365 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:58.498313 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hs2qb" podStartSLOduration=66.722500563 podStartE2EDuration="1m11.498298861s" totalImagesPullingTime="4.775798298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 16:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:58.497285619 +0000 UTC m=+72.078372190" watchObservedRunningTime="2026-07-26 16:09:58.498298861 +0000 UTC m=+72.079385431" Jul 26 16:09:58.514278 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:09:58.514216 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dndsz" podStartSLOduration=67.068334951 podStartE2EDuration="1m11.514202364s" totalImagesPullingTime="4.445867413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:58.513683203 +0000 UTC m=+72.094769774" watchObservedRunningTime="2026-07-26 16:09:58.514202364 +0000 UTC m=+72.095288935" Jul 26 16:10:09.425966 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.425821 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-599f4f87f4-25xgn" podUID="358697ed-6e24-42e2-9687-eee030fe181c" containerName="console" containerID="cri-o://d9a5e0fbb3da2aa3b003e18ed9fa0998564333843dc66a1c297c49524dd3a27c" gracePeriod=15 Jul 26 16:10:09.701084 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.701011 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:10:09.844055 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.844017 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/358697ed-6e24-42e2-9687-eee030fe181c-console-oauth-config\") pod \"358697ed-6e24-42e2-9687-eee030fe181c\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " Jul 26 16:10:09.844055 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.844060 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwjnn\" (UniqueName: \"kubernetes.io/projected/358697ed-6e24-42e2-9687-eee030fe181c-kube-api-access-hwjnn\") pod \"358697ed-6e24-42e2-9687-eee030fe181c\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " Jul 26 16:10:09.844306 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.844082 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/358697ed-6e24-42e2-9687-eee030fe181c-console-config\") pod \"358697ed-6e24-42e2-9687-eee030fe181c\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " Jul 26 16:10:09.844306 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.844161 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/358697ed-6e24-42e2-9687-eee030fe181c-oauth-serving-cert\") pod \"358697ed-6e24-42e2-9687-eee030fe181c\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " Jul 26 16:10:09.844306 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.844180 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/358697ed-6e24-42e2-9687-eee030fe181c-service-ca\") pod \"358697ed-6e24-42e2-9687-eee030fe181c\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " Jul 26 16:10:09.844306 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.844212 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/358697ed-6e24-42e2-9687-eee030fe181c-console-serving-cert\") pod \"358697ed-6e24-42e2-9687-eee030fe181c\" (UID: \"358697ed-6e24-42e2-9687-eee030fe181c\") " Jul 26 16:10:09.844622 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.844590 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358697ed-6e24-42e2-9687-eee030fe181c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "358697ed-6e24-42e2-9687-eee030fe181c" (UID: "358697ed-6e24-42e2-9687-eee030fe181c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:10:09.844701 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.844607 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358697ed-6e24-42e2-9687-eee030fe181c-console-config" (OuterVolumeSpecName: "console-config") pod "358697ed-6e24-42e2-9687-eee030fe181c" (UID: "358697ed-6e24-42e2-9687-eee030fe181c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:10:09.844701 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.844626 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358697ed-6e24-42e2-9687-eee030fe181c-service-ca" (OuterVolumeSpecName: "service-ca") pod "358697ed-6e24-42e2-9687-eee030fe181c" (UID: "358697ed-6e24-42e2-9687-eee030fe181c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:10:09.846508 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.846467 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358697ed-6e24-42e2-9687-eee030fe181c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "358697ed-6e24-42e2-9687-eee030fe181c" (UID: "358697ed-6e24-42e2-9687-eee030fe181c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:10:09.846607 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.846512 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358697ed-6e24-42e2-9687-eee030fe181c-kube-api-access-hwjnn" (OuterVolumeSpecName: "kube-api-access-hwjnn") pod "358697ed-6e24-42e2-9687-eee030fe181c" (UID: "358697ed-6e24-42e2-9687-eee030fe181c"). InnerVolumeSpecName "kube-api-access-hwjnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:10:09.846607 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.846520 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358697ed-6e24-42e2-9687-eee030fe181c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "358697ed-6e24-42e2-9687-eee030fe181c" (UID: "358697ed-6e24-42e2-9687-eee030fe181c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:10:09.944866 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.944822 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/358697ed-6e24-42e2-9687-eee030fe181c-oauth-serving-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:10:09.944866 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.944862 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/358697ed-6e24-42e2-9687-eee030fe181c-service-ca\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:10:09.945052 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.944877 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/358697ed-6e24-42e2-9687-eee030fe181c-console-serving-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:10:09.945052 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.944892 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/358697ed-6e24-42e2-9687-eee030fe181c-console-oauth-config\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:10:09.945052 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.944905 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwjnn\" (UniqueName: \"kubernetes.io/projected/358697ed-6e24-42e2-9687-eee030fe181c-kube-api-access-hwjnn\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:10:09.945052 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:09.944919 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/358697ed-6e24-42e2-9687-eee030fe181c-console-config\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:10:10.526530 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:10.526497 2563 generic.go:356] "Generic (PLEG): container finished" podID="358697ed-6e24-42e2-9687-eee030fe181c" containerID="d9a5e0fbb3da2aa3b003e18ed9fa0998564333843dc66a1c297c49524dd3a27c" exitCode=0 Jul 26 16:10:10.526970 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:10.526550 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599f4f87f4-25xgn" event={"ID":"358697ed-6e24-42e2-9687-eee030fe181c","Type":"ContainerDied","Data":"d9a5e0fbb3da2aa3b003e18ed9fa0998564333843dc66a1c297c49524dd3a27c"} Jul 26 16:10:10.526970 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:10.526555 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599f4f87f4-25xgn" Jul 26 16:10:10.526970 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:10.526614 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599f4f87f4-25xgn" event={"ID":"358697ed-6e24-42e2-9687-eee030fe181c","Type":"ContainerDied","Data":"84e19cf467504f2cc2161720fc207eeadc569a5bc23acd0fd9a218e046c7eee9"} Jul 26 16:10:10.526970 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:10.526644 2563 scope.go:117] "RemoveContainer" containerID="d9a5e0fbb3da2aa3b003e18ed9fa0998564333843dc66a1c297c49524dd3a27c" Jul 26 16:10:10.538292 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:10.538275 2563 scope.go:117] "RemoveContainer" containerID="d9a5e0fbb3da2aa3b003e18ed9fa0998564333843dc66a1c297c49524dd3a27c" Jul 26 16:10:10.538554 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:10:10.538530 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9a5e0fbb3da2aa3b003e18ed9fa0998564333843dc66a1c297c49524dd3a27c\": container with ID starting with d9a5e0fbb3da2aa3b003e18ed9fa0998564333843dc66a1c297c49524dd3a27c not found: ID does not exist" containerID="d9a5e0fbb3da2aa3b003e18ed9fa0998564333843dc66a1c297c49524dd3a27c" Jul 26 16:10:10.538622 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:10.538567 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9a5e0fbb3da2aa3b003e18ed9fa0998564333843dc66a1c297c49524dd3a27c"} err="failed to get container status \"d9a5e0fbb3da2aa3b003e18ed9fa0998564333843dc66a1c297c49524dd3a27c\": rpc error: code = NotFound desc = could not find container \"d9a5e0fbb3da2aa3b003e18ed9fa0998564333843dc66a1c297c49524dd3a27c\": container with ID starting with d9a5e0fbb3da2aa3b003e18ed9fa0998564333843dc66a1c297c49524dd3a27c not found: ID does not exist" Jul 26 16:10:10.548993 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:10.548966 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-599f4f87f4-25xgn"] Jul 26 16:10:10.555489 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:10.555468 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-599f4f87f4-25xgn"] Jul 26 16:10:10.889312 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:10.889211 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="358697ed-6e24-42e2-9687-eee030fe181c" path="/var/lib/kubelet/pods/358697ed-6e24-42e2-9687-eee030fe181c/volumes" Jul 26 16:10:21.546986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:21.546927 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74cf85ccf5-gzmp8" podUID="32d6e71f-f56f-4fc4-92e0-fed8eef0f686" containerName="console" containerID="cri-o://937ab77d7127a5962e8eacf69cb78b38bd69d2aa054e83fd6f690bbd933f7abc" gracePeriod=15 Jul 26 16:10:21.795256 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:21.795213 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:10:21.941376 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:21.941278 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-trusted-ca-bundle\") pod \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " Jul 26 16:10:21.941376 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:21.941321 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-console-oauth-config\") pod \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " Jul 26 16:10:21.941376 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:21.941346 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbxz6\" (UniqueName: \"kubernetes.io/projected/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-kube-api-access-fbxz6\") pod \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " Jul 26 16:10:21.941376 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:21.941361 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-oauth-serving-cert\") pod \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " Jul 26 16:10:21.941671 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:21.941387 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-console-config\") pod \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " Jul 26 16:10:21.941671 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:21.941451 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-service-ca\") pod \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " Jul 26 16:10:21.941671 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:21.941498 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-console-serving-cert\") pod \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\" (UID: \"32d6e71f-f56f-4fc4-92e0-fed8eef0f686\") " Jul 26 16:10:21.941900 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:21.941873 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "32d6e71f-f56f-4fc4-92e0-fed8eef0f686" (UID: "32d6e71f-f56f-4fc4-92e0-fed8eef0f686"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:10:21.941967 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:21.941898 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "32d6e71f-f56f-4fc4-92e0-fed8eef0f686" (UID: "32d6e71f-f56f-4fc4-92e0-fed8eef0f686"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:10:21.941967 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:21.941917 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-service-ca" (OuterVolumeSpecName: "service-ca") pod "32d6e71f-f56f-4fc4-92e0-fed8eef0f686" (UID: "32d6e71f-f56f-4fc4-92e0-fed8eef0f686"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:10:21.942127 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:21.942109 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-console-config" (OuterVolumeSpecName: "console-config") pod "32d6e71f-f56f-4fc4-92e0-fed8eef0f686" (UID: "32d6e71f-f56f-4fc4-92e0-fed8eef0f686"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:10:21.943793 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:21.943767 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "32d6e71f-f56f-4fc4-92e0-fed8eef0f686" (UID: "32d6e71f-f56f-4fc4-92e0-fed8eef0f686"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:10:21.944077 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:21.944051 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-kube-api-access-fbxz6" (OuterVolumeSpecName: "kube-api-access-fbxz6") pod "32d6e71f-f56f-4fc4-92e0-fed8eef0f686" (UID: "32d6e71f-f56f-4fc4-92e0-fed8eef0f686"). InnerVolumeSpecName "kube-api-access-fbxz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:10:21.944077 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:21.944058 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "32d6e71f-f56f-4fc4-92e0-fed8eef0f686" (UID: "32d6e71f-f56f-4fc4-92e0-fed8eef0f686"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:10:22.042603 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:22.042523 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-service-ca\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:10:22.042603 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:22.042594 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-console-serving-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:10:22.042603 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:22.042604 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-trusted-ca-bundle\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:10:22.042603 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:22.042614 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-console-oauth-config\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:10:22.042603 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:22.042623 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbxz6\" (UniqueName: \"kubernetes.io/projected/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-kube-api-access-fbxz6\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:10:22.042903 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:22.042632 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-oauth-serving-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:10:22.042903 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:22.042640 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32d6e71f-f56f-4fc4-92e0-fed8eef0f686-console-config\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:10:22.573203 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:22.573161 2563 generic.go:356] "Generic (PLEG): container finished" podID="32d6e71f-f56f-4fc4-92e0-fed8eef0f686" containerID="937ab77d7127a5962e8eacf69cb78b38bd69d2aa054e83fd6f690bbd933f7abc" exitCode=0 Jul 26 16:10:22.573203 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:22.573202 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74cf85ccf5-gzmp8" Jul 26 16:10:22.573701 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:22.573251 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74cf85ccf5-gzmp8" event={"ID":"32d6e71f-f56f-4fc4-92e0-fed8eef0f686","Type":"ContainerDied","Data":"937ab77d7127a5962e8eacf69cb78b38bd69d2aa054e83fd6f690bbd933f7abc"} Jul 26 16:10:22.573701 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:22.573298 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74cf85ccf5-gzmp8" event={"ID":"32d6e71f-f56f-4fc4-92e0-fed8eef0f686","Type":"ContainerDied","Data":"f90c2c26d6d0699803b0731338e9351101829b01a61619c59cc2e796e4838cb3"} Jul 26 16:10:22.573701 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:22.573318 2563 scope.go:117] "RemoveContainer" containerID="937ab77d7127a5962e8eacf69cb78b38bd69d2aa054e83fd6f690bbd933f7abc" Jul 26 16:10:22.585507 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:22.585481 2563 scope.go:117] "RemoveContainer" containerID="937ab77d7127a5962e8eacf69cb78b38bd69d2aa054e83fd6f690bbd933f7abc" Jul 26 16:10:22.585822 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:10:22.585800 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"937ab77d7127a5962e8eacf69cb78b38bd69d2aa054e83fd6f690bbd933f7abc\": container with ID starting with 937ab77d7127a5962e8eacf69cb78b38bd69d2aa054e83fd6f690bbd933f7abc not found: ID does not exist" containerID="937ab77d7127a5962e8eacf69cb78b38bd69d2aa054e83fd6f690bbd933f7abc" Jul 26 16:10:22.585903 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:22.585830 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"937ab77d7127a5962e8eacf69cb78b38bd69d2aa054e83fd6f690bbd933f7abc"} err="failed to get container status \"937ab77d7127a5962e8eacf69cb78b38bd69d2aa054e83fd6f690bbd933f7abc\": rpc error: code = NotFound desc = could not find container \"937ab77d7127a5962e8eacf69cb78b38bd69d2aa054e83fd6f690bbd933f7abc\": container with ID starting with 937ab77d7127a5962e8eacf69cb78b38bd69d2aa054e83fd6f690bbd933f7abc not found: ID does not exist" Jul 26 16:10:22.604439 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:22.604411 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74cf85ccf5-gzmp8"] Jul 26 16:10:22.608328 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:22.608303 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74cf85ccf5-gzmp8"] Jul 26 16:10:22.889311 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:22.889216 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32d6e71f-f56f-4fc4-92e0-fed8eef0f686" path="/var/lib/kubelet/pods/32d6e71f-f56f-4fc4-92e0-fed8eef0f686/volumes" Jul 26 16:10:29.490357 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:10:29.490327 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dndsz" Jul 26 16:11:03.466550 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.466513 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5b967cd444-tk24s"] Jul 26 16:11:03.467063 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.466872 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d6e71f-f56f-4fc4-92e0-fed8eef0f686" containerName="console" Jul 26 16:11:03.467063 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.466889 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d6e71f-f56f-4fc4-92e0-fed8eef0f686" containerName="console" Jul 26 16:11:03.467063 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.466913 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358697ed-6e24-42e2-9687-eee030fe181c" containerName="console" Jul 26 16:11:03.467063 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.466921 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="358697ed-6e24-42e2-9687-eee030fe181c" containerName="console" Jul 26 16:11:03.467063 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.466981 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="358697ed-6e24-42e2-9687-eee030fe181c" containerName="console" Jul 26 16:11:03.467063 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.466992 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32d6e71f-f56f-4fc4-92e0-fed8eef0f686" containerName="console" Jul 26 16:11:03.473467 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.473445 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.475964 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.475942 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 26 16:11:03.476078 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.475978 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 26 16:11:03.476078 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.475944 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 26 16:11:03.476295 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.476276 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 26 16:11:03.476295 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.476291 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-92frw\"" Jul 26 16:11:03.476486 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.476320 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 26 16:11:03.482487 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.482462 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 26 16:11:03.482725 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.482694 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b967cd444-tk24s"] Jul 26 16:11:03.536610 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.536564 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.536792 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.536622 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-serving-certs-ca-bundle\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.536792 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.536660 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-metrics-client-ca\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.536792 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.536680 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-federate-client-tls\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.536792 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.536700 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.536792 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.536732 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjlqg\" (UniqueName: \"kubernetes.io/projected/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-kube-api-access-xjlqg\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.536792 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.536757 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-telemeter-client-tls\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.537004 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.536794 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-secret-telemeter-client\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.637862 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.637829 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-telemeter-client-tls\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.637967 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.637867 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-secret-telemeter-client\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.637967 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.637890 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.637967 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.637917 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-serving-certs-ca-bundle\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.638117 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.638032 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-metrics-client-ca\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.638117 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.638082 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-federate-client-tls\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.638215 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.638119 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.638215 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.638171 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjlqg\" (UniqueName: \"kubernetes.io/projected/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-kube-api-access-xjlqg\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.638773 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.638747 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-serving-certs-ca-bundle\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.638886 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.638747 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-metrics-client-ca\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.639142 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.639113 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.640956 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.640933 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-secret-telemeter-client\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.641041 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.640977 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-telemeter-client-tls\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.641080 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.641039 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-federate-client-tls\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.641080 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.641042 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.648155 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.648130 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjlqg\" (UniqueName: \"kubernetes.io/projected/807200d8-49e0-4189-9ea5-4d4fd6b2e46c-kube-api-access-xjlqg\") pod \"telemeter-client-5b967cd444-tk24s\" (UID: \"807200d8-49e0-4189-9ea5-4d4fd6b2e46c\") " pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.784910 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.784871 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" Jul 26 16:11:03.912388 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:03.912353 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b967cd444-tk24s"] Jul 26 16:11:03.915246 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:11:03.915205 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod807200d8_49e0_4189_9ea5_4d4fd6b2e46c.slice/crio-bd297024719a39c362756c886e45e15c27543bce5de7f9201a62119230355017 WatchSource:0}: Error finding container bd297024719a39c362756c886e45e15c27543bce5de7f9201a62119230355017: Status 404 returned error can't find the container with id bd297024719a39c362756c886e45e15c27543bce5de7f9201a62119230355017 Jul 26 16:11:04.724819 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:04.724780 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" event={"ID":"807200d8-49e0-4189-9ea5-4d4fd6b2e46c","Type":"ContainerStarted","Data":"bd297024719a39c362756c886e45e15c27543bce5de7f9201a62119230355017"} Jul 26 16:11:05.730084 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:05.730048 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" event={"ID":"807200d8-49e0-4189-9ea5-4d4fd6b2e46c","Type":"ContainerStarted","Data":"35a6924d688bfaf99dc38cc319c4a9890e8679ef18988e3d913b1860ae849dfe"} Jul 26 16:11:05.730434 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:05.730118 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" event={"ID":"807200d8-49e0-4189-9ea5-4d4fd6b2e46c","Type":"ContainerStarted","Data":"debddb80f8968800c35333357e535939c96d5af0dc185b5fe48694f76e946c1c"} Jul 26 16:11:05.730434 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:05.730146 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" event={"ID":"807200d8-49e0-4189-9ea5-4d4fd6b2e46c","Type":"ContainerStarted","Data":"4343124a50faefa7a9a6a9cf3e1d5a61d08a748e4c8766236d9bc941fc029bc1"} Jul 26 16:11:05.752387 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:05.752295 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5b967cd444-tk24s" podStartSLOduration=1.167782861 podStartE2EDuration="2.752280251s" totalImagesPullingTime="1.58449739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:11:05.751887315 +0000 UTC m=+139.332973908" watchObservedRunningTime="2026-07-26 16:11:05.752280251 +0000 UTC m=+139.333366820" Jul 26 16:11:06.473065 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.473030 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bcb8ff49f-nfkvv"] Jul 26 16:11:06.476234 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.476208 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.486077 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.486055 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bcb8ff49f-nfkvv"] Jul 26 16:11:06.561531 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.561497 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9590bd5-df64-4f07-97ec-f999dc03cdb9-console-oauth-config\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.561531 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.561538 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-console-config\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.561753 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.561560 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9590bd5-df64-4f07-97ec-f999dc03cdb9-console-serving-cert\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.561753 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.561668 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-oauth-serving-cert\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.561753 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.561691 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9prx9\" (UniqueName: \"kubernetes.io/projected/b9590bd5-df64-4f07-97ec-f999dc03cdb9-kube-api-access-9prx9\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.561753 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.561723 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-trusted-ca-bundle\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.561753 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.561754 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-service-ca\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.662480 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.662440 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9590bd5-df64-4f07-97ec-f999dc03cdb9-console-oauth-config\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.662480 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.662485 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-console-config\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.662706 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.662506 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9590bd5-df64-4f07-97ec-f999dc03cdb9-console-serving-cert\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.662706 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.662537 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-oauth-serving-cert\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.662706 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.662557 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prx9\" (UniqueName: \"kubernetes.io/projected/b9590bd5-df64-4f07-97ec-f999dc03cdb9-kube-api-access-9prx9\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.662706 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.662592 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-trusted-ca-bundle\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.662706 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.662617 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-service-ca\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.663387 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.663362 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-oauth-serving-cert\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.663511 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.663362 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-service-ca\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.663511 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.663364 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-console-config\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.663605 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.663524 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-trusted-ca-bundle\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.664953 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.664926 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9590bd5-df64-4f07-97ec-f999dc03cdb9-console-oauth-config\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.665206 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.665114 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9590bd5-df64-4f07-97ec-f999dc03cdb9-console-serving-cert\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.669907 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.669887 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9prx9\" (UniqueName: \"kubernetes.io/projected/b9590bd5-df64-4f07-97ec-f999dc03cdb9-kube-api-access-9prx9\") pod \"console-5bcb8ff49f-nfkvv\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.784893 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.784868 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:06.909469 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:06.909438 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bcb8ff49f-nfkvv"] Jul 26 16:11:06.912133 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:11:06.912100 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9590bd5_df64_4f07_97ec_f999dc03cdb9.slice/crio-afd65962b44ed2878b54376a101c8985d38c6669b70b5cf48c3b7ebf34084c54 WatchSource:0}: Error finding container afd65962b44ed2878b54376a101c8985d38c6669b70b5cf48c3b7ebf34084c54: Status 404 returned error can't find the container with id afd65962b44ed2878b54376a101c8985d38c6669b70b5cf48c3b7ebf34084c54 Jul 26 16:11:07.738070 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:07.738036 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcb8ff49f-nfkvv" event={"ID":"b9590bd5-df64-4f07-97ec-f999dc03cdb9","Type":"ContainerStarted","Data":"69a83cef38d613bca2041f4701c60f2e18f1f787868bf49611e83ac941e913b6"} Jul 26 16:11:07.738277 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:07.738093 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcb8ff49f-nfkvv" event={"ID":"b9590bd5-df64-4f07-97ec-f999dc03cdb9","Type":"ContainerStarted","Data":"afd65962b44ed2878b54376a101c8985d38c6669b70b5cf48c3b7ebf34084c54"} Jul 26 16:11:07.756591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:07.756543 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bcb8ff49f-nfkvv" podStartSLOduration=1.75652678 podStartE2EDuration="1.75652678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:11:07.756109065 +0000 UTC m=+141.337195635" watchObservedRunningTime="2026-07-26 16:11:07.75652678 +0000 UTC m=+141.337613349" Jul 26 16:11:16.785495 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:16.785449 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:16.785971 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:16.785525 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:16.790201 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:16.790180 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:17.777585 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:17.777553 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:17.825607 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:17.825565 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9df5c9885-6snv6"] Jul 26 16:11:17.939055 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:17.939022 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69c9b49979-67cns"] Jul 26 16:11:17.942364 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:17.942345 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:17.951435 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:17.951407 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69c9b49979-67cns"] Jul 26 16:11:18.059209 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.059116 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brg6r\" (UniqueName: \"kubernetes.io/projected/81a471a5-49b8-416f-90a4-e5700cd96bdf-kube-api-access-brg6r\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.059209 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.059161 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-console-config\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.059209 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.059205 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81a471a5-49b8-416f-90a4-e5700cd96bdf-console-oauth-config\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.059494 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.059252 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-service-ca\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.059494 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.059270 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-oauth-serving-cert\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.059494 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.059317 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-trusted-ca-bundle\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.059494 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.059352 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81a471a5-49b8-416f-90a4-e5700cd96bdf-console-serving-cert\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.160116 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.160077 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81a471a5-49b8-416f-90a4-e5700cd96bdf-console-serving-cert\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.160317 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.160126 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brg6r\" (UniqueName: \"kubernetes.io/projected/81a471a5-49b8-416f-90a4-e5700cd96bdf-kube-api-access-brg6r\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.160317 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.160159 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-console-config\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.160317 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.160197 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81a471a5-49b8-416f-90a4-e5700cd96bdf-console-oauth-config\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.160317 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.160214 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-service-ca\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.160317 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.160259 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-oauth-serving-cert\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.160317 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.160290 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-trusted-ca-bundle\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.161072 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.161042 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-service-ca\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.161194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.161140 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-oauth-serving-cert\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.161194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.161182 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-trusted-ca-bundle\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.161342 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.161149 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-console-config\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.162723 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.162699 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81a471a5-49b8-416f-90a4-e5700cd96bdf-console-oauth-config\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.162806 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.162739 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81a471a5-49b8-416f-90a4-e5700cd96bdf-console-serving-cert\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.167865 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.167845 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brg6r\" (UniqueName: \"kubernetes.io/projected/81a471a5-49b8-416f-90a4-e5700cd96bdf-kube-api-access-brg6r\") pod \"console-69c9b49979-67cns\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.253278 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.253206 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:18.380052 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.380025 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69c9b49979-67cns"] Jul 26 16:11:18.382449 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:11:18.382417 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a471a5_49b8_416f_90a4_e5700cd96bdf.slice/crio-2d11e4e340b21cc90efe43a1535b08f039fb57464ac9f1a93ffb3906cd41477f WatchSource:0}: Error finding container 2d11e4e340b21cc90efe43a1535b08f039fb57464ac9f1a93ffb3906cd41477f: Status 404 returned error can't find the container with id 2d11e4e340b21cc90efe43a1535b08f039fb57464ac9f1a93ffb3906cd41477f Jul 26 16:11:18.779025 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.778993 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c9b49979-67cns" event={"ID":"81a471a5-49b8-416f-90a4-e5700cd96bdf","Type":"ContainerStarted","Data":"9f049d3fdb50eb2fbb136b79ef855f2676aae737225fb35eabef1735bfa6cded"} Jul 26 16:11:18.779247 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.779045 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c9b49979-67cns" event={"ID":"81a471a5-49b8-416f-90a4-e5700cd96bdf","Type":"ContainerStarted","Data":"2d11e4e340b21cc90efe43a1535b08f039fb57464ac9f1a93ffb3906cd41477f"} Jul 26 16:11:18.797194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:18.797144 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69c9b49979-67cns" podStartSLOduration=1.797128598 podStartE2EDuration="1.797128598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:11:18.795258656 +0000 UTC m=+152.376345232" watchObservedRunningTime="2026-07-26 16:11:18.797128598 +0000 UTC m=+152.378215168" Jul 26 16:11:28.253499 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:28.253459 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:28.254100 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:28.253527 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:28.258258 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:28.258235 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:28.820514 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:28.820488 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:11:28.861916 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:28.861882 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bcb8ff49f-nfkvv"] Jul 26 16:11:42.847757 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:42.847664 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9df5c9885-6snv6" podUID="6063ef00-8ee2-4b3a-8baa-d7027e0f562b" containerName="console" containerID="cri-o://d43f340138e4b1df1d5b77061c57f25b2619e2d505e64d847f303068bda71a94" gracePeriod=15 Jul 26 16:11:43.096206 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.096180 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:11:43.174554 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.174476 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-console-serving-cert\") pod \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " Jul 26 16:11:43.174554 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.174508 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-service-ca\") pod \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " Jul 26 16:11:43.174761 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.174561 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-console-oauth-config\") pod \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " Jul 26 16:11:43.174761 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.174580 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97jc5\" (UniqueName: \"kubernetes.io/projected/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-kube-api-access-97jc5\") pod \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " Jul 26 16:11:43.174761 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.174603 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-oauth-serving-cert\") pod \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " Jul 26 16:11:43.174761 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.174629 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-console-config\") pod \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " Jul 26 16:11:43.174761 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.174655 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-trusted-ca-bundle\") pod \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\" (UID: \"6063ef00-8ee2-4b3a-8baa-d7027e0f562b\") " Jul 26 16:11:43.175018 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.174976 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-service-ca" (OuterVolumeSpecName: "service-ca") pod "6063ef00-8ee2-4b3a-8baa-d7027e0f562b" (UID: "6063ef00-8ee2-4b3a-8baa-d7027e0f562b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:11:43.175018 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.174984 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6063ef00-8ee2-4b3a-8baa-d7027e0f562b" (UID: "6063ef00-8ee2-4b3a-8baa-d7027e0f562b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:11:43.175145 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.175116 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-console-config" (OuterVolumeSpecName: "console-config") pod "6063ef00-8ee2-4b3a-8baa-d7027e0f562b" (UID: "6063ef00-8ee2-4b3a-8baa-d7027e0f562b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:11:43.175187 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.175157 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6063ef00-8ee2-4b3a-8baa-d7027e0f562b" (UID: "6063ef00-8ee2-4b3a-8baa-d7027e0f562b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:11:43.176882 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.176853 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6063ef00-8ee2-4b3a-8baa-d7027e0f562b" (UID: "6063ef00-8ee2-4b3a-8baa-d7027e0f562b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:11:43.176987 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.176879 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-kube-api-access-97jc5" (OuterVolumeSpecName: "kube-api-access-97jc5") pod "6063ef00-8ee2-4b3a-8baa-d7027e0f562b" (UID: "6063ef00-8ee2-4b3a-8baa-d7027e0f562b"). InnerVolumeSpecName "kube-api-access-97jc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:11:43.176987 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.176893 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6063ef00-8ee2-4b3a-8baa-d7027e0f562b" (UID: "6063ef00-8ee2-4b3a-8baa-d7027e0f562b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:11:43.275600 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.275565 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-console-serving-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:11:43.275600 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.275594 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-service-ca\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:11:43.275600 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.275605 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-console-oauth-config\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:11:43.275826 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.275614 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97jc5\" (UniqueName: \"kubernetes.io/projected/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-kube-api-access-97jc5\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:11:43.275826 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.275625 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-oauth-serving-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:11:43.275826 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.275633 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-console-config\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:11:43.275826 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.275640 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6063ef00-8ee2-4b3a-8baa-d7027e0f562b-trusted-ca-bundle\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:11:43.870449 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.870410 2563 generic.go:356] "Generic (PLEG): container finished" podID="6063ef00-8ee2-4b3a-8baa-d7027e0f562b" containerID="d43f340138e4b1df1d5b77061c57f25b2619e2d505e64d847f303068bda71a94" exitCode=0 Jul 26 16:11:43.871026 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.870462 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9df5c9885-6snv6" Jul 26 16:11:43.871026 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.870499 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9df5c9885-6snv6" event={"ID":"6063ef00-8ee2-4b3a-8baa-d7027e0f562b","Type":"ContainerDied","Data":"d43f340138e4b1df1d5b77061c57f25b2619e2d505e64d847f303068bda71a94"} Jul 26 16:11:43.871026 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.870547 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9df5c9885-6snv6" event={"ID":"6063ef00-8ee2-4b3a-8baa-d7027e0f562b","Type":"ContainerDied","Data":"c9aa7f22072cbfcb093ba5eccf13e62ea8284b0d18273338f7a2ab24d8d5c231"} Jul 26 16:11:43.871026 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.870568 2563 scope.go:117] "RemoveContainer" containerID="d43f340138e4b1df1d5b77061c57f25b2619e2d505e64d847f303068bda71a94" Jul 26 16:11:43.879026 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.879008 2563 scope.go:117] "RemoveContainer" containerID="d43f340138e4b1df1d5b77061c57f25b2619e2d505e64d847f303068bda71a94" Jul 26 16:11:43.879292 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:11:43.879276 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d43f340138e4b1df1d5b77061c57f25b2619e2d505e64d847f303068bda71a94\": container with ID starting with d43f340138e4b1df1d5b77061c57f25b2619e2d505e64d847f303068bda71a94 not found: ID does not exist" containerID="d43f340138e4b1df1d5b77061c57f25b2619e2d505e64d847f303068bda71a94" Jul 26 16:11:43.879347 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.879301 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d43f340138e4b1df1d5b77061c57f25b2619e2d505e64d847f303068bda71a94"} err="failed to get container status \"d43f340138e4b1df1d5b77061c57f25b2619e2d505e64d847f303068bda71a94\": rpc error: code = NotFound desc = could not find container \"d43f340138e4b1df1d5b77061c57f25b2619e2d505e64d847f303068bda71a94\": container with ID starting with d43f340138e4b1df1d5b77061c57f25b2619e2d505e64d847f303068bda71a94 not found: ID does not exist" Jul 26 16:11:43.891436 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.891406 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9df5c9885-6snv6"] Jul 26 16:11:43.894100 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:43.894069 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9df5c9885-6snv6"] Jul 26 16:11:44.888856 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:44.888822 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6063ef00-8ee2-4b3a-8baa-d7027e0f562b" path="/var/lib/kubelet/pods/6063ef00-8ee2-4b3a-8baa-d7027e0f562b/volumes" Jul 26 16:11:53.881601 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:53.881555 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bcb8ff49f-nfkvv" podUID="b9590bd5-df64-4f07-97ec-f999dc03cdb9" containerName="console" containerID="cri-o://69a83cef38d613bca2041f4701c60f2e18f1f787868bf49611e83ac941e913b6" gracePeriod=15 Jul 26 16:11:54.117036 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.117011 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:54.159135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.159060 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-trusted-ca-bundle\") pod \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " Jul 26 16:11:54.159135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.159116 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9prx9\" (UniqueName: \"kubernetes.io/projected/b9590bd5-df64-4f07-97ec-f999dc03cdb9-kube-api-access-9prx9\") pod \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " Jul 26 16:11:54.159372 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.159144 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-oauth-serving-cert\") pod \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " Jul 26 16:11:54.159372 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.159273 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9590bd5-df64-4f07-97ec-f999dc03cdb9-console-serving-cert\") pod \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " Jul 26 16:11:54.159372 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.159348 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-console-config\") pod \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " Jul 26 16:11:54.159525 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.159392 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9590bd5-df64-4f07-97ec-f999dc03cdb9-console-oauth-config\") pod \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " Jul 26 16:11:54.159525 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.159423 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-service-ca\") pod \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\" (UID: \"b9590bd5-df64-4f07-97ec-f999dc03cdb9\") " Jul 26 16:11:54.159525 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.159513 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b9590bd5-df64-4f07-97ec-f999dc03cdb9" (UID: "b9590bd5-df64-4f07-97ec-f999dc03cdb9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:11:54.159678 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.159522 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b9590bd5-df64-4f07-97ec-f999dc03cdb9" (UID: "b9590bd5-df64-4f07-97ec-f999dc03cdb9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:11:54.159795 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.159767 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-console-config" (OuterVolumeSpecName: "console-config") pod "b9590bd5-df64-4f07-97ec-f999dc03cdb9" (UID: "b9590bd5-df64-4f07-97ec-f999dc03cdb9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:11:54.159853 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.159813 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-service-ca" (OuterVolumeSpecName: "service-ca") pod "b9590bd5-df64-4f07-97ec-f999dc03cdb9" (UID: "b9590bd5-df64-4f07-97ec-f999dc03cdb9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:11:54.159853 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.159828 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-trusted-ca-bundle\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:11:54.159922 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.159851 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-oauth-serving-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:11:54.161385 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.161364 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9590bd5-df64-4f07-97ec-f999dc03cdb9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b9590bd5-df64-4f07-97ec-f999dc03cdb9" (UID: "b9590bd5-df64-4f07-97ec-f999dc03cdb9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:11:54.161460 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.161416 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9590bd5-df64-4f07-97ec-f999dc03cdb9-kube-api-access-9prx9" (OuterVolumeSpecName: "kube-api-access-9prx9") pod "b9590bd5-df64-4f07-97ec-f999dc03cdb9" (UID: "b9590bd5-df64-4f07-97ec-f999dc03cdb9"). InnerVolumeSpecName "kube-api-access-9prx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:11:54.161496 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.161465 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9590bd5-df64-4f07-97ec-f999dc03cdb9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b9590bd5-df64-4f07-97ec-f999dc03cdb9" (UID: "b9590bd5-df64-4f07-97ec-f999dc03cdb9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:11:54.260541 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.260502 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-console-config\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:11:54.260541 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.260534 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9590bd5-df64-4f07-97ec-f999dc03cdb9-console-oauth-config\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:11:54.260541 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.260544 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9590bd5-df64-4f07-97ec-f999dc03cdb9-service-ca\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:11:54.260766 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.260553 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9prx9\" (UniqueName: \"kubernetes.io/projected/b9590bd5-df64-4f07-97ec-f999dc03cdb9-kube-api-access-9prx9\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:11:54.260766 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.260561 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9590bd5-df64-4f07-97ec-f999dc03cdb9-console-serving-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:11:54.911650 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.911615 2563 generic.go:356] "Generic (PLEG): container finished" podID="b9590bd5-df64-4f07-97ec-f999dc03cdb9" containerID="69a83cef38d613bca2041f4701c60f2e18f1f787868bf49611e83ac941e913b6" exitCode=0 Jul 26 16:11:54.912106 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.911659 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcb8ff49f-nfkvv" event={"ID":"b9590bd5-df64-4f07-97ec-f999dc03cdb9","Type":"ContainerDied","Data":"69a83cef38d613bca2041f4701c60f2e18f1f787868bf49611e83ac941e913b6"} Jul 26 16:11:54.912106 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.911664 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcb8ff49f-nfkvv" Jul 26 16:11:54.912106 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.911701 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcb8ff49f-nfkvv" event={"ID":"b9590bd5-df64-4f07-97ec-f999dc03cdb9","Type":"ContainerDied","Data":"afd65962b44ed2878b54376a101c8985d38c6669b70b5cf48c3b7ebf34084c54"} Jul 26 16:11:54.912106 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.911721 2563 scope.go:117] "RemoveContainer" containerID="69a83cef38d613bca2041f4701c60f2e18f1f787868bf49611e83ac941e913b6" Jul 26 16:11:54.919352 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.919336 2563 scope.go:117] "RemoveContainer" containerID="69a83cef38d613bca2041f4701c60f2e18f1f787868bf49611e83ac941e913b6" Jul 26 16:11:54.919593 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:11:54.919574 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a83cef38d613bca2041f4701c60f2e18f1f787868bf49611e83ac941e913b6\": container with ID starting with 69a83cef38d613bca2041f4701c60f2e18f1f787868bf49611e83ac941e913b6 not found: ID does not exist" containerID="69a83cef38d613bca2041f4701c60f2e18f1f787868bf49611e83ac941e913b6" Jul 26 16:11:54.919652 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.919601 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a83cef38d613bca2041f4701c60f2e18f1f787868bf49611e83ac941e913b6"} err="failed to get container status \"69a83cef38d613bca2041f4701c60f2e18f1f787868bf49611e83ac941e913b6\": rpc error: code = NotFound desc = could not find container \"69a83cef38d613bca2041f4701c60f2e18f1f787868bf49611e83ac941e913b6\": container with ID starting with 69a83cef38d613bca2041f4701c60f2e18f1f787868bf49611e83ac941e913b6 not found: ID does not exist" Jul 26 16:11:54.928617 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.928597 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bcb8ff49f-nfkvv"] Jul 26 16:11:54.939359 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:54.939334 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bcb8ff49f-nfkvv"] Jul 26 16:11:56.889281 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:11:56.889246 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9590bd5-df64-4f07-97ec-f999dc03cdb9" path="/var/lib/kubelet/pods/b9590bd5-df64-4f07-97ec-f999dc03cdb9/volumes" Jul 26 16:12:44.693390 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.693357 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx"] Jul 26 16:12:44.693838 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.693648 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6063ef00-8ee2-4b3a-8baa-d7027e0f562b" containerName="console" Jul 26 16:12:44.693838 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.693658 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6063ef00-8ee2-4b3a-8baa-d7027e0f562b" containerName="console" Jul 26 16:12:44.693838 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.693668 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9590bd5-df64-4f07-97ec-f999dc03cdb9" containerName="console" Jul 26 16:12:44.693838 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.693673 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9590bd5-df64-4f07-97ec-f999dc03cdb9" containerName="console" Jul 26 16:12:44.693838 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.693717 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6063ef00-8ee2-4b3a-8baa-d7027e0f562b" containerName="console" Jul 26 16:12:44.693838 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.693723 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9590bd5-df64-4f07-97ec-f999dc03cdb9" containerName="console" Jul 26 16:12:44.696649 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.696632 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx" Jul 26 16:12:44.699698 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.699673 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 26 16:12:44.699807 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.699740 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jfd5b\"" Jul 26 16:12:44.700905 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.700890 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 26 16:12:44.706772 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.706751 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx"] Jul 26 16:12:44.838037 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.837998 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whkcc\" (UniqueName: \"kubernetes.io/projected/1ca50602-8453-463d-bd16-391fc9194c3a-kube-api-access-whkcc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx\" (UID: \"1ca50602-8453-463d-bd16-391fc9194c3a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx" Jul 26 16:12:44.838037 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.838041 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ca50602-8453-463d-bd16-391fc9194c3a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx\" (UID: \"1ca50602-8453-463d-bd16-391fc9194c3a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx" Jul 26 16:12:44.838288 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.838067 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ca50602-8453-463d-bd16-391fc9194c3a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx\" (UID: \"1ca50602-8453-463d-bd16-391fc9194c3a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx" Jul 26 16:12:44.939177 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.939138 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whkcc\" (UniqueName: \"kubernetes.io/projected/1ca50602-8453-463d-bd16-391fc9194c3a-kube-api-access-whkcc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx\" (UID: \"1ca50602-8453-463d-bd16-391fc9194c3a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx" Jul 26 16:12:44.939177 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.939187 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ca50602-8453-463d-bd16-391fc9194c3a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx\" (UID: \"1ca50602-8453-463d-bd16-391fc9194c3a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx" Jul 26 16:12:44.939410 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.939240 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ca50602-8453-463d-bd16-391fc9194c3a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx\" (UID: \"1ca50602-8453-463d-bd16-391fc9194c3a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx" Jul 26 16:12:44.939578 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.939559 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ca50602-8453-463d-bd16-391fc9194c3a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx\" (UID: \"1ca50602-8453-463d-bd16-391fc9194c3a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx" Jul 26 16:12:44.939622 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.939612 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ca50602-8453-463d-bd16-391fc9194c3a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx\" (UID: \"1ca50602-8453-463d-bd16-391fc9194c3a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx" Jul 26 16:12:44.948153 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:44.948088 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whkcc\" (UniqueName: \"kubernetes.io/projected/1ca50602-8453-463d-bd16-391fc9194c3a-kube-api-access-whkcc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx\" (UID: \"1ca50602-8453-463d-bd16-391fc9194c3a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx" Jul 26 16:12:45.005607 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:45.005578 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx" Jul 26 16:12:45.127861 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:45.127831 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx"] Jul 26 16:12:45.131008 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:12:45.130983 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca50602_8453_463d_bd16_391fc9194c3a.slice/crio-2d4b891b600f48f0774614e14ac7e07eed5f87684c7304a5c48dd8e8d5dc900d WatchSource:0}: Error finding container 2d4b891b600f48f0774614e14ac7e07eed5f87684c7304a5c48dd8e8d5dc900d: Status 404 returned error can't find the container with id 2d4b891b600f48f0774614e14ac7e07eed5f87684c7304a5c48dd8e8d5dc900d Jul 26 16:12:46.093309 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:46.093263 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx" event={"ID":"1ca50602-8453-463d-bd16-391fc9194c3a","Type":"ContainerStarted","Data":"2d4b891b600f48f0774614e14ac7e07eed5f87684c7304a5c48dd8e8d5dc900d"} Jul 26 16:12:53.121513 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:53.121471 2563 generic.go:356] "Generic (PLEG): container finished" podID="1ca50602-8453-463d-bd16-391fc9194c3a" containerID="2ebad827cb029a9d476249bc0a08b8bf5a0cb9e58e7dc20167c3b1e46b161188" exitCode=0 Jul 26 16:12:53.122016 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:53.121533 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx" event={"ID":"1ca50602-8453-463d-bd16-391fc9194c3a","Type":"ContainerDied","Data":"2ebad827cb029a9d476249bc0a08b8bf5a0cb9e58e7dc20167c3b1e46b161188"} Jul 26 16:12:56.133927 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:56.133885 2563 generic.go:356] "Generic (PLEG): container finished" podID="1ca50602-8453-463d-bd16-391fc9194c3a" containerID="8f8d69e0f38acf195fd61bbb1a9bc3a5d951363185c1dc9bc167cb9ce7be7442" exitCode=0 Jul 26 16:12:56.134344 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:12:56.133979 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx" event={"ID":"1ca50602-8453-463d-bd16-391fc9194c3a","Type":"ContainerDied","Data":"8f8d69e0f38acf195fd61bbb1a9bc3a5d951363185c1dc9bc167cb9ce7be7442"} Jul 26 16:13:04.170540 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:04.170501 2563 generic.go:356] "Generic (PLEG): container finished" podID="1ca50602-8453-463d-bd16-391fc9194c3a" containerID="640800584cbbbaf00155aea5c16efd0a68e650270aeec90f6420c7fedc5e4733" exitCode=0 Jul 26 16:13:04.170908 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:04.170590 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx" event={"ID":"1ca50602-8453-463d-bd16-391fc9194c3a","Type":"ContainerDied","Data":"640800584cbbbaf00155aea5c16efd0a68e650270aeec90f6420c7fedc5e4733"} Jul 26 16:13:05.301342 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:05.301320 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx" Jul 26 16:13:05.396972 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:05.396937 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whkcc\" (UniqueName: \"kubernetes.io/projected/1ca50602-8453-463d-bd16-391fc9194c3a-kube-api-access-whkcc\") pod \"1ca50602-8453-463d-bd16-391fc9194c3a\" (UID: \"1ca50602-8453-463d-bd16-391fc9194c3a\") " Jul 26 16:13:05.397127 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:05.396981 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ca50602-8453-463d-bd16-391fc9194c3a-util\") pod \"1ca50602-8453-463d-bd16-391fc9194c3a\" (UID: \"1ca50602-8453-463d-bd16-391fc9194c3a\") " Jul 26 16:13:05.399087 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:05.399057 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca50602-8453-463d-bd16-391fc9194c3a-kube-api-access-whkcc" (OuterVolumeSpecName: "kube-api-access-whkcc") pod "1ca50602-8453-463d-bd16-391fc9194c3a" (UID: "1ca50602-8453-463d-bd16-391fc9194c3a"). InnerVolumeSpecName "kube-api-access-whkcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:13:05.401394 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:05.401367 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca50602-8453-463d-bd16-391fc9194c3a-util" (OuterVolumeSpecName: "util") pod "1ca50602-8453-463d-bd16-391fc9194c3a" (UID: "1ca50602-8453-463d-bd16-391fc9194c3a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:13:05.498189 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:05.498106 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ca50602-8453-463d-bd16-391fc9194c3a-bundle\") pod \"1ca50602-8453-463d-bd16-391fc9194c3a\" (UID: \"1ca50602-8453-463d-bd16-391fc9194c3a\") " Jul 26 16:13:05.498360 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:05.498342 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whkcc\" (UniqueName: \"kubernetes.io/projected/1ca50602-8453-463d-bd16-391fc9194c3a-kube-api-access-whkcc\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:13:05.498405 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:05.498366 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ca50602-8453-463d-bd16-391fc9194c3a-util\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:13:05.498793 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:05.498776 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca50602-8453-463d-bd16-391fc9194c3a-bundle" (OuterVolumeSpecName: "bundle") pod "1ca50602-8453-463d-bd16-391fc9194c3a" (UID: "1ca50602-8453-463d-bd16-391fc9194c3a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:13:05.599229 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:05.599187 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ca50602-8453-463d-bd16-391fc9194c3a-bundle\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:13:06.180056 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:06.180029 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx" Jul 26 16:13:06.180243 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:06.180028 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mg2nx" event={"ID":"1ca50602-8453-463d-bd16-391fc9194c3a","Type":"ContainerDied","Data":"2d4b891b600f48f0774614e14ac7e07eed5f87684c7304a5c48dd8e8d5dc900d"} Jul 26 16:13:06.180243 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:06.180163 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d4b891b600f48f0774614e14ac7e07eed5f87684c7304a5c48dd8e8d5dc900d" Jul 26 16:13:12.670933 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.670847 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cpwlg"] Jul 26 16:13:12.671497 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.671250 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca50602-8453-463d-bd16-391fc9194c3a" containerName="pull" Jul 26 16:13:12.671497 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.671268 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca50602-8453-463d-bd16-391fc9194c3a" containerName="pull" Jul 26 16:13:12.671497 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.671285 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca50602-8453-463d-bd16-391fc9194c3a" containerName="extract" Jul 26 16:13:12.671497 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.671294 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca50602-8453-463d-bd16-391fc9194c3a" containerName="extract" Jul 26 16:13:12.671497 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.671309 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca50602-8453-463d-bd16-391fc9194c3a" containerName="util" Jul 26 16:13:12.671497 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.671318 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca50602-8453-463d-bd16-391fc9194c3a" containerName="util" Jul 26 16:13:12.671497 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.671395 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ca50602-8453-463d-bd16-391fc9194c3a" containerName="extract" Jul 26 16:13:12.673982 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.673963 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cpwlg" Jul 26 16:13:12.676721 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.676700 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 26 16:13:12.676721 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.676710 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 26 16:13:12.676906 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.676764 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9p9xt\"" Jul 26 16:13:12.684258 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.684235 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cpwlg"] Jul 26 16:13:12.747798 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.747761 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ggm8\" (UniqueName: \"kubernetes.io/projected/73def870-045c-41a3-b734-59d768f684c9-kube-api-access-5ggm8\") pod \"cert-manager-operator-controller-manager-55c444bf86-cpwlg\" (UID: \"73def870-045c-41a3-b734-59d768f684c9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cpwlg" Jul 26 16:13:12.747959 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.747807 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73def870-045c-41a3-b734-59d768f684c9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cpwlg\" (UID: \"73def870-045c-41a3-b734-59d768f684c9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cpwlg" Jul 26 16:13:12.848492 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.848447 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ggm8\" (UniqueName: \"kubernetes.io/projected/73def870-045c-41a3-b734-59d768f684c9-kube-api-access-5ggm8\") pod \"cert-manager-operator-controller-manager-55c444bf86-cpwlg\" (UID: \"73def870-045c-41a3-b734-59d768f684c9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cpwlg" Jul 26 16:13:12.848492 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.848503 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73def870-045c-41a3-b734-59d768f684c9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cpwlg\" (UID: \"73def870-045c-41a3-b734-59d768f684c9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cpwlg" Jul 26 16:13:12.848860 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.848843 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73def870-045c-41a3-b734-59d768f684c9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cpwlg\" (UID: \"73def870-045c-41a3-b734-59d768f684c9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cpwlg" Jul 26 16:13:12.857743 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.857714 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ggm8\" (UniqueName: \"kubernetes.io/projected/73def870-045c-41a3-b734-59d768f684c9-kube-api-access-5ggm8\") pod \"cert-manager-operator-controller-manager-55c444bf86-cpwlg\" (UID: \"73def870-045c-41a3-b734-59d768f684c9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cpwlg" Jul 26 16:13:12.983358 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:12.983263 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cpwlg" Jul 26 16:13:13.130938 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:13.130901 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cpwlg"] Jul 26 16:13:13.135604 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:13:13.135568 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73def870_045c_41a3_b734_59d768f684c9.slice/crio-8b448a9ab90aa64525f0ae864c0edda599d46703c7133409d5d721c916e9d865 WatchSource:0}: Error finding container 8b448a9ab90aa64525f0ae864c0edda599d46703c7133409d5d721c916e9d865: Status 404 returned error can't find the container with id 8b448a9ab90aa64525f0ae864c0edda599d46703c7133409d5d721c916e9d865 Jul 26 16:13:13.206194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:13.206162 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cpwlg" event={"ID":"73def870-045c-41a3-b734-59d768f684c9","Type":"ContainerStarted","Data":"8b448a9ab90aa64525f0ae864c0edda599d46703c7133409d5d721c916e9d865"} Jul 26 16:13:15.216894 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:15.216857 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cpwlg" event={"ID":"73def870-045c-41a3-b734-59d768f684c9","Type":"ContainerStarted","Data":"1b5829931c01ba18beaaec3a6e6dee62b24aaa9fba304358a95c860177782bfd"} Jul 26 16:13:15.251885 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:15.251820 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cpwlg" podStartSLOduration=1.9161408930000001 podStartE2EDuration="3.251795334s" totalImagesPullingTime="1.335654441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:13:15.249890501 +0000 UTC m=+268.830977073" watchObservedRunningTime="2026-07-26 16:13:15.251795334 +0000 UTC m=+268.832881905" Jul 26 16:13:15.622794 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:15.622760 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wct27"] Jul 26 16:13:15.626132 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:15.626112 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wct27" Jul 26 16:13:15.628910 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:15.628889 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 26 16:13:15.629022 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:15.628930 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-ljhw5\"" Jul 26 16:13:15.629207 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:15.629194 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 26 16:13:15.638797 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:15.638776 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wct27"] Jul 26 16:13:15.670073 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:15.670044 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltt62\" (UniqueName: \"kubernetes.io/projected/a5c9cad8-10bc-45f7-9d48-4bfae5c7f469-kube-api-access-ltt62\") pod \"cert-manager-webhook-6486ffd997-wct27\" (UID: \"a5c9cad8-10bc-45f7-9d48-4bfae5c7f469\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wct27" Jul 26 16:13:15.670189 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:15.670091 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5c9cad8-10bc-45f7-9d48-4bfae5c7f469-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wct27\" (UID: \"a5c9cad8-10bc-45f7-9d48-4bfae5c7f469\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wct27" Jul 26 16:13:15.770733 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:15.770698 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5c9cad8-10bc-45f7-9d48-4bfae5c7f469-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wct27\" (UID: \"a5c9cad8-10bc-45f7-9d48-4bfae5c7f469\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wct27" Jul 26 16:13:15.770898 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:15.770751 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltt62\" (UniqueName: \"kubernetes.io/projected/a5c9cad8-10bc-45f7-9d48-4bfae5c7f469-kube-api-access-ltt62\") pod \"cert-manager-webhook-6486ffd997-wct27\" (UID: \"a5c9cad8-10bc-45f7-9d48-4bfae5c7f469\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wct27" Jul 26 16:13:15.779483 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:15.779453 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5c9cad8-10bc-45f7-9d48-4bfae5c7f469-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wct27\" (UID: \"a5c9cad8-10bc-45f7-9d48-4bfae5c7f469\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wct27" Jul 26 16:13:15.785586 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:15.785560 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltt62\" (UniqueName: \"kubernetes.io/projected/a5c9cad8-10bc-45f7-9d48-4bfae5c7f469-kube-api-access-ltt62\") pod \"cert-manager-webhook-6486ffd997-wct27\" (UID: \"a5c9cad8-10bc-45f7-9d48-4bfae5c7f469\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wct27" Jul 26 16:13:15.946678 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:15.946589 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wct27" Jul 26 16:13:16.071164 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:16.071139 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wct27"] Jul 26 16:13:16.073378 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:13:16.073352 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5c9cad8_10bc_45f7_9d48_4bfae5c7f469.slice/crio-52fad023589fd12a4c8e0ee78a3726d7afab5e22b340711de82f278ae2b74a41 WatchSource:0}: Error finding container 52fad023589fd12a4c8e0ee78a3726d7afab5e22b340711de82f278ae2b74a41: Status 404 returned error can't find the container with id 52fad023589fd12a4c8e0ee78a3726d7afab5e22b340711de82f278ae2b74a41 Jul 26 16:13:16.222441 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:16.222348 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wct27" event={"ID":"a5c9cad8-10bc-45f7-9d48-4bfae5c7f469","Type":"ContainerStarted","Data":"52fad023589fd12a4c8e0ee78a3726d7afab5e22b340711de82f278ae2b74a41"} Jul 26 16:13:16.944374 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:16.941748 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk"] Jul 26 16:13:16.947696 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:16.947661 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk" Jul 26 16:13:16.950485 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:16.950457 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 26 16:13:16.950619 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:16.950564 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 26 16:13:16.950619 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:16.950468 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jfd5b\"" Jul 26 16:13:16.951015 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:16.950993 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk"] Jul 26 16:13:16.980592 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:16.980567 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgtz7\" (UniqueName: \"kubernetes.io/projected/ed0a16be-36d0-42d9-af60-46d18bb44aba-kube-api-access-pgtz7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk\" (UID: \"ed0a16be-36d0-42d9-af60-46d18bb44aba\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk" Jul 26 16:13:16.980725 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:16.980601 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed0a16be-36d0-42d9-af60-46d18bb44aba-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk\" (UID: \"ed0a16be-36d0-42d9-af60-46d18bb44aba\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk" Jul 26 16:13:16.980725 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:16.980628 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed0a16be-36d0-42d9-af60-46d18bb44aba-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk\" (UID: \"ed0a16be-36d0-42d9-af60-46d18bb44aba\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk" Jul 26 16:13:17.081957 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:17.081916 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgtz7\" (UniqueName: \"kubernetes.io/projected/ed0a16be-36d0-42d9-af60-46d18bb44aba-kube-api-access-pgtz7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk\" (UID: \"ed0a16be-36d0-42d9-af60-46d18bb44aba\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk" Jul 26 16:13:17.081957 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:17.081960 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed0a16be-36d0-42d9-af60-46d18bb44aba-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk\" (UID: \"ed0a16be-36d0-42d9-af60-46d18bb44aba\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk" Jul 26 16:13:17.082147 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:17.082077 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed0a16be-36d0-42d9-af60-46d18bb44aba-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk\" (UID: \"ed0a16be-36d0-42d9-af60-46d18bb44aba\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk" Jul 26 16:13:17.082396 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:17.082376 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed0a16be-36d0-42d9-af60-46d18bb44aba-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk\" (UID: \"ed0a16be-36d0-42d9-af60-46d18bb44aba\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk" Jul 26 16:13:17.082439 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:17.082414 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed0a16be-36d0-42d9-af60-46d18bb44aba-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk\" (UID: \"ed0a16be-36d0-42d9-af60-46d18bb44aba\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk" Jul 26 16:13:17.091017 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:17.090998 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgtz7\" (UniqueName: \"kubernetes.io/projected/ed0a16be-36d0-42d9-af60-46d18bb44aba-kube-api-access-pgtz7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk\" (UID: \"ed0a16be-36d0-42d9-af60-46d18bb44aba\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk" Jul 26 16:13:17.257704 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:17.257622 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk" Jul 26 16:13:17.409073 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:17.409048 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk"] Jul 26 16:13:17.412630 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:13:17.412584 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0a16be_36d0_42d9_af60_46d18bb44aba.slice/crio-bb0038ead42abe6f6cb719b84521888ef40943a2a77fe3c58470170bbd9090da WatchSource:0}: Error finding container bb0038ead42abe6f6cb719b84521888ef40943a2a77fe3c58470170bbd9090da: Status 404 returned error can't find the container with id bb0038ead42abe6f6cb719b84521888ef40943a2a77fe3c58470170bbd9090da Jul 26 16:13:18.232362 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:18.232326 2563 generic.go:356] "Generic (PLEG): container finished" podID="ed0a16be-36d0-42d9-af60-46d18bb44aba" containerID="fd4ec42eae67c3f5fd019d8c7833bc5501b0f3840576f169122085e46110caee" exitCode=0 Jul 26 16:13:18.232531 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:18.232411 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk" event={"ID":"ed0a16be-36d0-42d9-af60-46d18bb44aba","Type":"ContainerDied","Data":"fd4ec42eae67c3f5fd019d8c7833bc5501b0f3840576f169122085e46110caee"} Jul 26 16:13:18.232531 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:18.232482 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk" event={"ID":"ed0a16be-36d0-42d9-af60-46d18bb44aba","Type":"ContainerStarted","Data":"bb0038ead42abe6f6cb719b84521888ef40943a2a77fe3c58470170bbd9090da"} Jul 26 16:13:20.243434 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:20.243394 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wct27" event={"ID":"a5c9cad8-10bc-45f7-9d48-4bfae5c7f469","Type":"ContainerStarted","Data":"377311c10823771b031d22b820becb38d27886dc6c559833aae7282ae5de647f"} Jul 26 16:13:20.243855 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:20.243549 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wct27" Jul 26 16:13:20.259342 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:20.259279 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-wct27" podStartSLOduration=2.126432689 podStartE2EDuration="5.259258738s" totalImagesPullingTime="3.132826049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:13:20.258738335 +0000 UTC m=+273.839824909" watchObservedRunningTime="2026-07-26 16:13:20.259258738 +0000 UTC m=+273.840345308" Jul 26 16:13:20.607651 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:20.607620 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-j8xqq"] Jul 26 16:13:20.610666 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:20.610647 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j8xqq" Jul 26 16:13:20.613326 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:20.613298 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-dxmgw\"" Jul 26 16:13:20.622062 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:20.622039 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-j8xqq"] Jul 26 16:13:20.713893 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:20.713851 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf1eb834-bacd-46d3-a186-5bec334f5546-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-j8xqq\" (UID: \"bf1eb834-bacd-46d3-a186-5bec334f5546\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j8xqq" Jul 26 16:13:20.714040 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:20.713920 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xcbw\" (UniqueName: \"kubernetes.io/projected/bf1eb834-bacd-46d3-a186-5bec334f5546-kube-api-access-7xcbw\") pod \"cert-manager-cainjector-559cdd8c87-j8xqq\" (UID: \"bf1eb834-bacd-46d3-a186-5bec334f5546\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j8xqq" Jul 26 16:13:20.814471 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:20.814433 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf1eb834-bacd-46d3-a186-5bec334f5546-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-j8xqq\" (UID: \"bf1eb834-bacd-46d3-a186-5bec334f5546\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j8xqq" Jul 26 16:13:20.814622 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:20.814495 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xcbw\" (UniqueName: \"kubernetes.io/projected/bf1eb834-bacd-46d3-a186-5bec334f5546-kube-api-access-7xcbw\") pod \"cert-manager-cainjector-559cdd8c87-j8xqq\" (UID: \"bf1eb834-bacd-46d3-a186-5bec334f5546\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j8xqq" Jul 26 16:13:20.824946 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:20.824918 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf1eb834-bacd-46d3-a186-5bec334f5546-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-j8xqq\" (UID: \"bf1eb834-bacd-46d3-a186-5bec334f5546\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j8xqq" Jul 26 16:13:20.825075 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:20.824989 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xcbw\" (UniqueName: \"kubernetes.io/projected/bf1eb834-bacd-46d3-a186-5bec334f5546-kube-api-access-7xcbw\") pod \"cert-manager-cainjector-559cdd8c87-j8xqq\" (UID: \"bf1eb834-bacd-46d3-a186-5bec334f5546\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j8xqq" Jul 26 16:13:20.919844 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:20.919769 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j8xqq" Jul 26 16:13:21.048691 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:21.048665 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-j8xqq"] Jul 26 16:13:21.051012 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:13:21.050980 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf1eb834_bacd_46d3_a186_5bec334f5546.slice/crio-32b6247ef04207d041b7e60b56085f80bcaa1aa2bc2a1ec30f7d94b9f21e2be7 WatchSource:0}: Error finding container 32b6247ef04207d041b7e60b56085f80bcaa1aa2bc2a1ec30f7d94b9f21e2be7: Status 404 returned error can't find the container with id 32b6247ef04207d041b7e60b56085f80bcaa1aa2bc2a1ec30f7d94b9f21e2be7 Jul 26 16:13:21.249652 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:21.249556 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j8xqq" event={"ID":"bf1eb834-bacd-46d3-a186-5bec334f5546","Type":"ContainerStarted","Data":"a3594cdbc9b070bc8829c27a47e707593da7bec3eec2851ba3f61f58d785aca7"} Jul 26 16:13:21.249652 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:21.249609 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j8xqq" event={"ID":"bf1eb834-bacd-46d3-a186-5bec334f5546","Type":"ContainerStarted","Data":"32b6247ef04207d041b7e60b56085f80bcaa1aa2bc2a1ec30f7d94b9f21e2be7"} Jul 26 16:13:21.251618 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:21.251581 2563 generic.go:356] "Generic (PLEG): container finished" podID="ed0a16be-36d0-42d9-af60-46d18bb44aba" containerID="7e925017e14d3318cf46071931f02bcf0bf5ccc8adbb3aba6515ff503508575e" exitCode=0 Jul 26 16:13:21.251723 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:21.251626 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk" event={"ID":"ed0a16be-36d0-42d9-af60-46d18bb44aba","Type":"ContainerDied","Data":"7e925017e14d3318cf46071931f02bcf0bf5ccc8adbb3aba6515ff503508575e"} Jul 26 16:13:21.266819 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:21.266767 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j8xqq" podStartSLOduration=1.266750473 podStartE2EDuration="1.266750473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:13:21.266607023 +0000 UTC m=+274.847693597" watchObservedRunningTime="2026-07-26 16:13:21.266750473 +0000 UTC m=+274.847837044" Jul 26 16:13:22.258969 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:22.258935 2563 generic.go:356] "Generic (PLEG): container finished" podID="ed0a16be-36d0-42d9-af60-46d18bb44aba" containerID="1a4f52aa54befb109b4a331026b76423e0fea7f881827b8191d0ec69d505a5ab" exitCode=0 Jul 26 16:13:22.259390 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:22.259024 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk" event={"ID":"ed0a16be-36d0-42d9-af60-46d18bb44aba","Type":"ContainerDied","Data":"1a4f52aa54befb109b4a331026b76423e0fea7f881827b8191d0ec69d505a5ab"} Jul 26 16:13:23.387558 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:23.387265 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk" Jul 26 16:13:23.438043 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:23.438013 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed0a16be-36d0-42d9-af60-46d18bb44aba-bundle\") pod \"ed0a16be-36d0-42d9-af60-46d18bb44aba\" (UID: \"ed0a16be-36d0-42d9-af60-46d18bb44aba\") " Jul 26 16:13:23.438235 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:23.438075 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed0a16be-36d0-42d9-af60-46d18bb44aba-util\") pod \"ed0a16be-36d0-42d9-af60-46d18bb44aba\" (UID: \"ed0a16be-36d0-42d9-af60-46d18bb44aba\") " Jul 26 16:13:23.438235 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:23.438107 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgtz7\" (UniqueName: \"kubernetes.io/projected/ed0a16be-36d0-42d9-af60-46d18bb44aba-kube-api-access-pgtz7\") pod \"ed0a16be-36d0-42d9-af60-46d18bb44aba\" (UID: \"ed0a16be-36d0-42d9-af60-46d18bb44aba\") " Jul 26 16:13:23.438517 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:23.438480 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0a16be-36d0-42d9-af60-46d18bb44aba-bundle" (OuterVolumeSpecName: "bundle") pod "ed0a16be-36d0-42d9-af60-46d18bb44aba" (UID: "ed0a16be-36d0-42d9-af60-46d18bb44aba"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:13:23.440226 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:23.440198 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0a16be-36d0-42d9-af60-46d18bb44aba-kube-api-access-pgtz7" (OuterVolumeSpecName: "kube-api-access-pgtz7") pod "ed0a16be-36d0-42d9-af60-46d18bb44aba" (UID: "ed0a16be-36d0-42d9-af60-46d18bb44aba"). InnerVolumeSpecName "kube-api-access-pgtz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:13:23.441977 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:23.441939 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0a16be-36d0-42d9-af60-46d18bb44aba-util" (OuterVolumeSpecName: "util") pod "ed0a16be-36d0-42d9-af60-46d18bb44aba" (UID: "ed0a16be-36d0-42d9-af60-46d18bb44aba"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:13:23.539271 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:23.539237 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgtz7\" (UniqueName: \"kubernetes.io/projected/ed0a16be-36d0-42d9-af60-46d18bb44aba-kube-api-access-pgtz7\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:13:23.539271 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:23.539265 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed0a16be-36d0-42d9-af60-46d18bb44aba-bundle\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:13:23.539271 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:23.539276 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed0a16be-36d0-42d9-af60-46d18bb44aba-util\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:13:24.270069 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:24.270037 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk" event={"ID":"ed0a16be-36d0-42d9-af60-46d18bb44aba","Type":"ContainerDied","Data":"bb0038ead42abe6f6cb719b84521888ef40943a2a77fe3c58470170bbd9090da"} Jul 26 16:13:24.270069 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:24.270070 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44mdk" Jul 26 16:13:24.270280 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:24.270085 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb0038ead42abe6f6cb719b84521888ef40943a2a77fe3c58470170bbd9090da" Jul 26 16:13:26.254046 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:26.254015 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wct27" Jul 26 16:13:27.058526 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.058489 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bdv9b"] Jul 26 16:13:27.058774 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.058762 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed0a16be-36d0-42d9-af60-46d18bb44aba" containerName="extract" Jul 26 16:13:27.058819 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.058775 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed0a16be-36d0-42d9-af60-46d18bb44aba" containerName="extract" Jul 26 16:13:27.058819 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.058786 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed0a16be-36d0-42d9-af60-46d18bb44aba" containerName="util" Jul 26 16:13:27.058819 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.058791 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed0a16be-36d0-42d9-af60-46d18bb44aba" containerName="util" Jul 26 16:13:27.058819 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.058810 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed0a16be-36d0-42d9-af60-46d18bb44aba" containerName="pull" Jul 26 16:13:27.058819 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.058815 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed0a16be-36d0-42d9-af60-46d18bb44aba" containerName="pull" Jul 26 16:13:27.058964 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.058861 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed0a16be-36d0-42d9-af60-46d18bb44aba" containerName="extract" Jul 26 16:13:27.062638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.062617 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bdv9b" Jul 26 16:13:27.065350 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.065330 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-75mm9\"" Jul 26 16:13:27.071639 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.071617 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bdv9b"] Jul 26 16:13:27.165997 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.165960 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe80c73a-929d-4e6b-b54e-867158d9bb9a-bound-sa-token\") pod \"cert-manager-55b4d954d5-bdv9b\" (UID: \"fe80c73a-929d-4e6b-b54e-867158d9bb9a\") " pod="cert-manager/cert-manager-55b4d954d5-bdv9b" Jul 26 16:13:27.166165 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.166019 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5cqk\" (UniqueName: \"kubernetes.io/projected/fe80c73a-929d-4e6b-b54e-867158d9bb9a-kube-api-access-k5cqk\") pod \"cert-manager-55b4d954d5-bdv9b\" (UID: \"fe80c73a-929d-4e6b-b54e-867158d9bb9a\") " pod="cert-manager/cert-manager-55b4d954d5-bdv9b" Jul 26 16:13:27.267276 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.267242 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe80c73a-929d-4e6b-b54e-867158d9bb9a-bound-sa-token\") pod \"cert-manager-55b4d954d5-bdv9b\" (UID: \"fe80c73a-929d-4e6b-b54e-867158d9bb9a\") " pod="cert-manager/cert-manager-55b4d954d5-bdv9b" Jul 26 16:13:27.267659 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.267314 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5cqk\" (UniqueName: \"kubernetes.io/projected/fe80c73a-929d-4e6b-b54e-867158d9bb9a-kube-api-access-k5cqk\") pod \"cert-manager-55b4d954d5-bdv9b\" (UID: \"fe80c73a-929d-4e6b-b54e-867158d9bb9a\") " pod="cert-manager/cert-manager-55b4d954d5-bdv9b" Jul 26 16:13:27.279023 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.278984 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe80c73a-929d-4e6b-b54e-867158d9bb9a-bound-sa-token\") pod \"cert-manager-55b4d954d5-bdv9b\" (UID: \"fe80c73a-929d-4e6b-b54e-867158d9bb9a\") " pod="cert-manager/cert-manager-55b4d954d5-bdv9b" Jul 26 16:13:27.279175 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.279089 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5cqk\" (UniqueName: \"kubernetes.io/projected/fe80c73a-929d-4e6b-b54e-867158d9bb9a-kube-api-access-k5cqk\") pod \"cert-manager-55b4d954d5-bdv9b\" (UID: \"fe80c73a-929d-4e6b-b54e-867158d9bb9a\") " pod="cert-manager/cert-manager-55b4d954d5-bdv9b" Jul 26 16:13:27.372183 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.372101 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bdv9b" Jul 26 16:13:27.499932 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:27.499900 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bdv9b"] Jul 26 16:13:27.502588 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:13:27.502557 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe80c73a_929d_4e6b_b54e_867158d9bb9a.slice/crio-cd918dfcb1adfe4464f0ac188c7710e5f26a1c719ac9e3319707e95aef2eb046 WatchSource:0}: Error finding container cd918dfcb1adfe4464f0ac188c7710e5f26a1c719ac9e3319707e95aef2eb046: Status 404 returned error can't find the container with id cd918dfcb1adfe4464f0ac188c7710e5f26a1c719ac9e3319707e95aef2eb046 Jul 26 16:13:28.287957 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:28.287921 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bdv9b" event={"ID":"fe80c73a-929d-4e6b-b54e-867158d9bb9a","Type":"ContainerStarted","Data":"a1b36af9342025dba024c0de07be0aa3d04ed9bd5966a2ece284d7e13329201c"} Jul 26 16:13:28.288455 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:28.287971 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bdv9b" event={"ID":"fe80c73a-929d-4e6b-b54e-867158d9bb9a","Type":"ContainerStarted","Data":"cd918dfcb1adfe4464f0ac188c7710e5f26a1c719ac9e3319707e95aef2eb046"} Jul 26 16:13:28.306523 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:28.306475 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-bdv9b" podStartSLOduration=1.306458171 podStartE2EDuration="1.306458171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:13:28.305594791 +0000 UTC m=+281.886681362" watchObservedRunningTime="2026-07-26 16:13:28.306458171 +0000 UTC m=+281.887544741" Jul 26 16:13:41.205338 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:41.205299 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x"] Jul 26 16:13:41.207471 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:41.207455 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x" Jul 26 16:13:41.210204 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:41.210185 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 26 16:13:41.210319 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:41.210213 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jfd5b\"" Jul 26 16:13:41.210319 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:41.210236 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 26 16:13:41.217464 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:41.217443 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x"] Jul 26 16:13:41.282377 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:41.282343 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwsxw\" (UniqueName: \"kubernetes.io/projected/c5610e24-1a13-423b-b5c0-e96d0bd1b801-kube-api-access-wwsxw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x\" (UID: \"c5610e24-1a13-423b-b5c0-e96d0bd1b801\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x" Jul 26 16:13:41.282545 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:41.282388 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5610e24-1a13-423b-b5c0-e96d0bd1b801-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x\" (UID: \"c5610e24-1a13-423b-b5c0-e96d0bd1b801\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x" Jul 26 16:13:41.282545 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:41.282479 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5610e24-1a13-423b-b5c0-e96d0bd1b801-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x\" (UID: \"c5610e24-1a13-423b-b5c0-e96d0bd1b801\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x" Jul 26 16:13:41.383195 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:41.383161 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwsxw\" (UniqueName: \"kubernetes.io/projected/c5610e24-1a13-423b-b5c0-e96d0bd1b801-kube-api-access-wwsxw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x\" (UID: \"c5610e24-1a13-423b-b5c0-e96d0bd1b801\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x" Jul 26 16:13:41.383390 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:41.383207 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5610e24-1a13-423b-b5c0-e96d0bd1b801-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x\" (UID: \"c5610e24-1a13-423b-b5c0-e96d0bd1b801\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x" Jul 26 16:13:41.383390 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:41.383294 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5610e24-1a13-423b-b5c0-e96d0bd1b801-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x\" (UID: \"c5610e24-1a13-423b-b5c0-e96d0bd1b801\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x" Jul 26 16:13:41.383635 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:41.383614 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5610e24-1a13-423b-b5c0-e96d0bd1b801-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x\" (UID: \"c5610e24-1a13-423b-b5c0-e96d0bd1b801\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x" Jul 26 16:13:41.383713 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:41.383673 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5610e24-1a13-423b-b5c0-e96d0bd1b801-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x\" (UID: \"c5610e24-1a13-423b-b5c0-e96d0bd1b801\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x" Jul 26 16:13:41.391155 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:41.391121 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwsxw\" (UniqueName: \"kubernetes.io/projected/c5610e24-1a13-423b-b5c0-e96d0bd1b801-kube-api-access-wwsxw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x\" (UID: \"c5610e24-1a13-423b-b5c0-e96d0bd1b801\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x" Jul 26 16:13:41.518499 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:41.518405 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x" Jul 26 16:13:41.650036 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:41.650000 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x"] Jul 26 16:13:41.653967 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:13:41.653939 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5610e24_1a13_423b_b5c0_e96d0bd1b801.slice/crio-40a3773ea85dcc1718b5e10ec7acb4aa31ff60f5bd5e16dd3d3be072f3a83aa2 WatchSource:0}: Error finding container 40a3773ea85dcc1718b5e10ec7acb4aa31ff60f5bd5e16dd3d3be072f3a83aa2: Status 404 returned error can't find the container with id 40a3773ea85dcc1718b5e10ec7acb4aa31ff60f5bd5e16dd3d3be072f3a83aa2 Jul 26 16:13:42.350944 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:42.350907 2563 generic.go:356] "Generic (PLEG): container finished" podID="c5610e24-1a13-423b-b5c0-e96d0bd1b801" containerID="0bfbc3cd02e3091af974492da99d8e6146e5cf74f348c0474634a5120d601293" exitCode=0 Jul 26 16:13:42.351395 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:42.350999 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x" event={"ID":"c5610e24-1a13-423b-b5c0-e96d0bd1b801","Type":"ContainerDied","Data":"0bfbc3cd02e3091af974492da99d8e6146e5cf74f348c0474634a5120d601293"} Jul 26 16:13:42.351395 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:42.351101 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x" event={"ID":"c5610e24-1a13-423b-b5c0-e96d0bd1b801","Type":"ContainerStarted","Data":"40a3773ea85dcc1718b5e10ec7acb4aa31ff60f5bd5e16dd3d3be072f3a83aa2"} Jul 26 16:13:43.356695 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:43.356656 2563 generic.go:356] "Generic (PLEG): container finished" podID="c5610e24-1a13-423b-b5c0-e96d0bd1b801" containerID="7142df833bf1006fdb0e58c2a2c600c9adbf9a08c7a8badc5eb259ff5ef7b7eb" exitCode=0 Jul 26 16:13:43.357167 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:43.356741 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x" event={"ID":"c5610e24-1a13-423b-b5c0-e96d0bd1b801","Type":"ContainerDied","Data":"7142df833bf1006fdb0e58c2a2c600c9adbf9a08c7a8badc5eb259ff5ef7b7eb"} Jul 26 16:13:44.365362 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:44.365331 2563 generic.go:356] "Generic (PLEG): container finished" podID="c5610e24-1a13-423b-b5c0-e96d0bd1b801" containerID="b08d55673c03c97184f77b04a06bfdb2bc12b6588059a0293c1b3abb1a7d1826" exitCode=0 Jul 26 16:13:44.365759 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:44.365414 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x" event={"ID":"c5610e24-1a13-423b-b5c0-e96d0bd1b801","Type":"ContainerDied","Data":"b08d55673c03c97184f77b04a06bfdb2bc12b6588059a0293c1b3abb1a7d1826"} Jul 26 16:13:45.498165 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:45.498137 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x" Jul 26 16:13:45.619833 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:45.619750 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5610e24-1a13-423b-b5c0-e96d0bd1b801-util\") pod \"c5610e24-1a13-423b-b5c0-e96d0bd1b801\" (UID: \"c5610e24-1a13-423b-b5c0-e96d0bd1b801\") " Jul 26 16:13:45.619833 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:45.619832 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5610e24-1a13-423b-b5c0-e96d0bd1b801-bundle\") pod \"c5610e24-1a13-423b-b5c0-e96d0bd1b801\" (UID: \"c5610e24-1a13-423b-b5c0-e96d0bd1b801\") " Jul 26 16:13:45.620008 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:45.619873 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwsxw\" (UniqueName: \"kubernetes.io/projected/c5610e24-1a13-423b-b5c0-e96d0bd1b801-kube-api-access-wwsxw\") pod \"c5610e24-1a13-423b-b5c0-e96d0bd1b801\" (UID: \"c5610e24-1a13-423b-b5c0-e96d0bd1b801\") " Jul 26 16:13:45.620517 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:45.620488 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5610e24-1a13-423b-b5c0-e96d0bd1b801-bundle" (OuterVolumeSpecName: "bundle") pod "c5610e24-1a13-423b-b5c0-e96d0bd1b801" (UID: "c5610e24-1a13-423b-b5c0-e96d0bd1b801"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:13:45.621921 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:45.621895 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5610e24-1a13-423b-b5c0-e96d0bd1b801-kube-api-access-wwsxw" (OuterVolumeSpecName: "kube-api-access-wwsxw") pod "c5610e24-1a13-423b-b5c0-e96d0bd1b801" (UID: "c5610e24-1a13-423b-b5c0-e96d0bd1b801"). InnerVolumeSpecName "kube-api-access-wwsxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:13:45.625923 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:45.625904 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5610e24-1a13-423b-b5c0-e96d0bd1b801-util" (OuterVolumeSpecName: "util") pod "c5610e24-1a13-423b-b5c0-e96d0bd1b801" (UID: "c5610e24-1a13-423b-b5c0-e96d0bd1b801"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:13:45.721069 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:45.721026 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwsxw\" (UniqueName: \"kubernetes.io/projected/c5610e24-1a13-423b-b5c0-e96d0bd1b801-kube-api-access-wwsxw\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:13:45.721069 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:45.721065 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5610e24-1a13-423b-b5c0-e96d0bd1b801-util\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:13:45.721069 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:45.721077 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5610e24-1a13-423b-b5c0-e96d0bd1b801-bundle\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:13:46.375617 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:46.375591 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x" Jul 26 16:13:46.375790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:46.375587 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv4w9x" event={"ID":"c5610e24-1a13-423b-b5c0-e96d0bd1b801","Type":"ContainerDied","Data":"40a3773ea85dcc1718b5e10ec7acb4aa31ff60f5bd5e16dd3d3be072f3a83aa2"} Jul 26 16:13:46.375790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:46.375709 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40a3773ea85dcc1718b5e10ec7acb4aa31ff60f5bd5e16dd3d3be072f3a83aa2" Jul 26 16:13:46.784940 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:46.784917 2563 kubelet.go:1648] "Image garbage collection succeeded" Jul 26 16:13:54.110907 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.110875 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6"] Jul 26 16:13:54.113330 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.111169 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5610e24-1a13-423b-b5c0-e96d0bd1b801" containerName="util" Jul 26 16:13:54.113330 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.111179 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5610e24-1a13-423b-b5c0-e96d0bd1b801" containerName="util" Jul 26 16:13:54.113330 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.111188 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5610e24-1a13-423b-b5c0-e96d0bd1b801" containerName="pull" Jul 26 16:13:54.113330 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.111195 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5610e24-1a13-423b-b5c0-e96d0bd1b801" containerName="pull" Jul 26 16:13:54.113330 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.111203 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5610e24-1a13-423b-b5c0-e96d0bd1b801" containerName="extract" Jul 26 16:13:54.113330 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.111209 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5610e24-1a13-423b-b5c0-e96d0bd1b801" containerName="extract" Jul 26 16:13:54.113330 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.111265 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5610e24-1a13-423b-b5c0-e96d0bd1b801" containerName="extract" Jul 26 16:13:54.114152 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.114136 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:13:54.117495 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.117470 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 26 16:13:54.118261 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.117859 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 26 16:13:54.118261 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.117910 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 26 16:13:54.118261 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.117859 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lg5vj\"" Jul 26 16:13:54.118261 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.118114 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 26 16:13:54.128366 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.128345 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6"] Jul 26 16:13:54.197235 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.197193 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5b87e136-eb08-4ca1-92f6-1b6e7d722248-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bd4986cdb-kg7w6\" (UID: \"5b87e136-eb08-4ca1-92f6-1b6e7d722248\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:13:54.197431 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.197272 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b87e136-eb08-4ca1-92f6-1b6e7d722248-tmp\") pod \"opendatahub-operator-controller-manager-7bd4986cdb-kg7w6\" (UID: \"5b87e136-eb08-4ca1-92f6-1b6e7d722248\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:13:54.197431 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.197292 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b87e136-eb08-4ca1-92f6-1b6e7d722248-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bd4986cdb-kg7w6\" (UID: \"5b87e136-eb08-4ca1-92f6-1b6e7d722248\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:13:54.197431 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.197317 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5b87e136-eb08-4ca1-92f6-1b6e7d722248-manifests\") pod \"opendatahub-operator-controller-manager-7bd4986cdb-kg7w6\" (UID: \"5b87e136-eb08-4ca1-92f6-1b6e7d722248\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:13:54.197431 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.197350 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wt92\" (UniqueName: \"kubernetes.io/projected/5b87e136-eb08-4ca1-92f6-1b6e7d722248-kube-api-access-6wt92\") pod \"opendatahub-operator-controller-manager-7bd4986cdb-kg7w6\" (UID: \"5b87e136-eb08-4ca1-92f6-1b6e7d722248\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:13:54.298563 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.298525 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5b87e136-eb08-4ca1-92f6-1b6e7d722248-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bd4986cdb-kg7w6\" (UID: \"5b87e136-eb08-4ca1-92f6-1b6e7d722248\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:13:54.298759 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.298595 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b87e136-eb08-4ca1-92f6-1b6e7d722248-tmp\") pod \"opendatahub-operator-controller-manager-7bd4986cdb-kg7w6\" (UID: \"5b87e136-eb08-4ca1-92f6-1b6e7d722248\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:13:54.298759 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.298623 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b87e136-eb08-4ca1-92f6-1b6e7d722248-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bd4986cdb-kg7w6\" (UID: \"5b87e136-eb08-4ca1-92f6-1b6e7d722248\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:13:54.298759 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.298655 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5b87e136-eb08-4ca1-92f6-1b6e7d722248-manifests\") pod \"opendatahub-operator-controller-manager-7bd4986cdb-kg7w6\" (UID: \"5b87e136-eb08-4ca1-92f6-1b6e7d722248\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:13:54.298759 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.298679 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wt92\" (UniqueName: \"kubernetes.io/projected/5b87e136-eb08-4ca1-92f6-1b6e7d722248-kube-api-access-6wt92\") pod \"opendatahub-operator-controller-manager-7bd4986cdb-kg7w6\" (UID: \"5b87e136-eb08-4ca1-92f6-1b6e7d722248\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:13:54.299060 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.299031 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5b87e136-eb08-4ca1-92f6-1b6e7d722248-manifests\") pod \"opendatahub-operator-controller-manager-7bd4986cdb-kg7w6\" (UID: \"5b87e136-eb08-4ca1-92f6-1b6e7d722248\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:13:54.300834 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.300812 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b87e136-eb08-4ca1-92f6-1b6e7d722248-tmp\") pod \"opendatahub-operator-controller-manager-7bd4986cdb-kg7w6\" (UID: \"5b87e136-eb08-4ca1-92f6-1b6e7d722248\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:13:54.301087 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.301069 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b87e136-eb08-4ca1-92f6-1b6e7d722248-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bd4986cdb-kg7w6\" (UID: \"5b87e136-eb08-4ca1-92f6-1b6e7d722248\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:13:54.301148 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.301105 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5b87e136-eb08-4ca1-92f6-1b6e7d722248-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bd4986cdb-kg7w6\" (UID: \"5b87e136-eb08-4ca1-92f6-1b6e7d722248\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:13:54.310535 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.310502 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wt92\" (UniqueName: \"kubernetes.io/projected/5b87e136-eb08-4ca1-92f6-1b6e7d722248-kube-api-access-6wt92\") pod \"opendatahub-operator-controller-manager-7bd4986cdb-kg7w6\" (UID: \"5b87e136-eb08-4ca1-92f6-1b6e7d722248\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:13:54.426331 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.426255 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:13:54.486051 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.486020 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5"] Jul 26 16:13:54.490962 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.490935 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" Jul 26 16:13:54.493982 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.493816 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 26 16:13:54.493982 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.493857 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 26 16:13:54.493982 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.493883 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 26 16:13:54.494588 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.494339 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 26 16:13:54.494588 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.494437 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-87vm7\"" Jul 26 16:13:54.494588 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.494340 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 26 16:13:54.504751 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.504730 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5"] Jul 26 16:13:54.588602 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.588572 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6"] Jul 26 16:13:54.591955 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:13:54.591932 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b87e136_eb08_4ca1_92f6_1b6e7d722248.slice/crio-6c906f3efb3499a3603ee3506615e2255c78873414e3b83e904d8e88c2902388 WatchSource:0}: Error finding container 6c906f3efb3499a3603ee3506615e2255c78873414e3b83e904d8e88c2902388: Status 404 returned error can't find the container with id 6c906f3efb3499a3603ee3506615e2255c78873414e3b83e904d8e88c2902388 Jul 26 16:13:54.593727 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.593711 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 16:13:54.600508 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.600486 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tdmr\" (UniqueName: \"kubernetes.io/projected/48d6169c-7e2b-4578-a4cc-b5ab9e6a9292-kube-api-access-7tdmr\") pod \"lws-controller-manager-76c849bdf9-8cqr5\" (UID: \"48d6169c-7e2b-4578-a4cc-b5ab9e6a9292\") " pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" Jul 26 16:13:54.600588 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.600535 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/48d6169c-7e2b-4578-a4cc-b5ab9e6a9292-manager-config\") pod \"lws-controller-manager-76c849bdf9-8cqr5\" (UID: \"48d6169c-7e2b-4578-a4cc-b5ab9e6a9292\") " pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" Jul 26 16:13:54.600588 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.600559 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48d6169c-7e2b-4578-a4cc-b5ab9e6a9292-cert\") pod \"lws-controller-manager-76c849bdf9-8cqr5\" (UID: \"48d6169c-7e2b-4578-a4cc-b5ab9e6a9292\") " pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" Jul 26 16:13:54.600652 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.600602 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48d6169c-7e2b-4578-a4cc-b5ab9e6a9292-metrics-cert\") pod \"lws-controller-manager-76c849bdf9-8cqr5\" (UID: \"48d6169c-7e2b-4578-a4cc-b5ab9e6a9292\") " pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" Jul 26 16:13:54.701842 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.701758 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48d6169c-7e2b-4578-a4cc-b5ab9e6a9292-metrics-cert\") pod \"lws-controller-manager-76c849bdf9-8cqr5\" (UID: \"48d6169c-7e2b-4578-a4cc-b5ab9e6a9292\") " pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" Jul 26 16:13:54.701842 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.701808 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tdmr\" (UniqueName: \"kubernetes.io/projected/48d6169c-7e2b-4578-a4cc-b5ab9e6a9292-kube-api-access-7tdmr\") pod \"lws-controller-manager-76c849bdf9-8cqr5\" (UID: \"48d6169c-7e2b-4578-a4cc-b5ab9e6a9292\") " pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" Jul 26 16:13:54.702024 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.701849 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/48d6169c-7e2b-4578-a4cc-b5ab9e6a9292-manager-config\") pod \"lws-controller-manager-76c849bdf9-8cqr5\" (UID: \"48d6169c-7e2b-4578-a4cc-b5ab9e6a9292\") " pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" Jul 26 16:13:54.702024 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.701879 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48d6169c-7e2b-4578-a4cc-b5ab9e6a9292-cert\") pod \"lws-controller-manager-76c849bdf9-8cqr5\" (UID: \"48d6169c-7e2b-4578-a4cc-b5ab9e6a9292\") " pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" Jul 26 16:13:54.702619 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.702601 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/48d6169c-7e2b-4578-a4cc-b5ab9e6a9292-manager-config\") pod \"lws-controller-manager-76c849bdf9-8cqr5\" (UID: \"48d6169c-7e2b-4578-a4cc-b5ab9e6a9292\") " pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" Jul 26 16:13:54.704271 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.704254 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48d6169c-7e2b-4578-a4cc-b5ab9e6a9292-cert\") pod \"lws-controller-manager-76c849bdf9-8cqr5\" (UID: \"48d6169c-7e2b-4578-a4cc-b5ab9e6a9292\") " pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" Jul 26 16:13:54.704327 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.704300 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48d6169c-7e2b-4578-a4cc-b5ab9e6a9292-metrics-cert\") pod \"lws-controller-manager-76c849bdf9-8cqr5\" (UID: \"48d6169c-7e2b-4578-a4cc-b5ab9e6a9292\") " pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" Jul 26 16:13:54.710404 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.710379 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tdmr\" (UniqueName: \"kubernetes.io/projected/48d6169c-7e2b-4578-a4cc-b5ab9e6a9292-kube-api-access-7tdmr\") pod \"lws-controller-manager-76c849bdf9-8cqr5\" (UID: \"48d6169c-7e2b-4578-a4cc-b5ab9e6a9292\") " pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" Jul 26 16:13:54.800526 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.800491 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" Jul 26 16:13:54.946293 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:54.946265 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5"] Jul 26 16:13:54.948410 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:13:54.948387 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d6169c_7e2b_4578_a4cc_b5ab9e6a9292.slice/crio-cdeeec41a19e2f6fbebe66c5ed1cf40abbd9ba1137d50368bbe564309b1aa5f6 WatchSource:0}: Error finding container cdeeec41a19e2f6fbebe66c5ed1cf40abbd9ba1137d50368bbe564309b1aa5f6: Status 404 returned error can't find the container with id cdeeec41a19e2f6fbebe66c5ed1cf40abbd9ba1137d50368bbe564309b1aa5f6 Jul 26 16:13:55.416805 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:55.416764 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" event={"ID":"48d6169c-7e2b-4578-a4cc-b5ab9e6a9292","Type":"ContainerStarted","Data":"cdeeec41a19e2f6fbebe66c5ed1cf40abbd9ba1137d50368bbe564309b1aa5f6"} Jul 26 16:13:55.419650 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:55.419611 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" event={"ID":"5b87e136-eb08-4ca1-92f6-1b6e7d722248","Type":"ContainerStarted","Data":"6c906f3efb3499a3603ee3506615e2255c78873414e3b83e904d8e88c2902388"} Jul 26 16:13:58.435661 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:58.435553 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" event={"ID":"48d6169c-7e2b-4578-a4cc-b5ab9e6a9292","Type":"ContainerStarted","Data":"4074e17215c3cb8be0ff7a3e6c1d07ff109d5893d3ae66d170a47abb52488fc4"} Jul 26 16:13:58.436082 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:58.435676 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" Jul 26 16:13:58.437044 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:58.437020 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" event={"ID":"5b87e136-eb08-4ca1-92f6-1b6e7d722248","Type":"ContainerStarted","Data":"438e1d831e98e21cd1330dfb97070786d5b3e7ef695e6680878f6d98800cf7ed"} Jul 26 16:13:58.463185 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:58.463125 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" podStartSLOduration=1.3218239569999999 podStartE2EDuration="4.463105059s" totalImagesPullingTime="3.141281102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:13:58.462048817 +0000 UTC m=+312.043135387" watchObservedRunningTime="2026-07-26 16:13:58.463105059 +0000 UTC m=+312.044191628" Jul 26 16:13:59.443216 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:59.443174 2563 generic.go:356] "Generic (PLEG): container finished" podID="5b87e136-eb08-4ca1-92f6-1b6e7d722248" containerID="438e1d831e98e21cd1330dfb97070786d5b3e7ef695e6680878f6d98800cf7ed" exitCode=0 Jul 26 16:13:59.443652 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:13:59.443217 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" event={"ID":"5b87e136-eb08-4ca1-92f6-1b6e7d722248","Type":"ContainerDied","Data":"438e1d831e98e21cd1330dfb97070786d5b3e7ef695e6680878f6d98800cf7ed"} Jul 26 16:14:00.449578 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:00.449540 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" event={"ID":"5b87e136-eb08-4ca1-92f6-1b6e7d722248","Type":"ContainerStarted","Data":"48af04f1d34255cf26774b7598fcbe87fe26b5469d26e71ec2dbd1436c96b7af"} Jul 26 16:14:00.449940 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:00.449629 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:14:00.474550 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:00.474487 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" podStartSLOduration=1.505142863 podStartE2EDuration="6.474464679s" totalImagesPullingTime="3.870099841s" totalInitContainerRuntime="1.099221975s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:13:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:14:00.470796358 +0000 UTC m=+314.051882928" watchObservedRunningTime="2026-07-26 16:14:00.474464679 +0000 UTC m=+314.055551250" Jul 26 16:14:09.445503 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:09.445472 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-76c849bdf9-8cqr5" Jul 26 16:14:11.457010 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:11.456976 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7bd4986cdb-kg7w6" Jul 26 16:14:20.870784 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:20.870752 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj"] Jul 26 16:14:20.877007 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:20.876984 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:20.880175 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:20.880148 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 26 16:14:20.880326 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:20.880273 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 26 16:14:20.880385 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:20.880273 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-k76qx\"" Jul 26 16:14:20.881295 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:20.881275 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 26 16:14:20.890521 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:20.890498 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj"] Jul 26 16:14:20.916814 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:20.916784 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:20.916959 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:20.916818 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:20.916959 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:20.916843 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a037c20-6177-4eea-ab0c-beef73b93bf6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:20.916959 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:20.916864 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:20.916959 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:20.916931 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:20.917144 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:20.916994 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:20.917144 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:20.917050 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:20.917144 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:20.917070 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:20.917144 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:20.917091 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s2xj\" (UniqueName: \"kubernetes.io/projected/9a037c20-6177-4eea-ab0c-beef73b93bf6-kube-api-access-9s2xj\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.018022 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.017932 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.018022 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.018019 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.018333 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.018056 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.018333 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.018098 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s2xj\" (UniqueName: \"kubernetes.io/projected/9a037c20-6177-4eea-ab0c-beef73b93bf6-kube-api-access-9s2xj\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.018333 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.018166 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.018333 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.018230 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.018333 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.018273 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a037c20-6177-4eea-ab0c-beef73b93bf6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.018333 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.018303 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.018769 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.018361 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.018769 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.018447 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.018769 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.018728 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.019271 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.018984 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.019271 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.019012 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.019271 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.019155 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a037c20-6177-4eea-ab0c-beef73b93bf6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.021415 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.021392 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.021762 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.021691 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.028641 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.028611 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.029859 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.029830 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s2xj\" (UniqueName: \"kubernetes.io/projected/9a037c20-6177-4eea-ab0c-beef73b93bf6-kube-api-access-9s2xj\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.137971 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.137890 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj"] Jul 26 16:14:21.141285 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.141266 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.156175 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.156138 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj"] Jul 26 16:14:21.188363 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.188321 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:21.220533 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.220493 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c09d668-aa2a-494f-902c-2c53c190bfc6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.220726 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.220586 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c09d668-aa2a-494f-902c-2c53c190bfc6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.220726 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.220635 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c09d668-aa2a-494f-902c-2c53c190bfc6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.220726 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.220655 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c09d668-aa2a-494f-902c-2c53c190bfc6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.220726 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.220688 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c09d668-aa2a-494f-902c-2c53c190bfc6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.220944 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.220736 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c09d668-aa2a-494f-902c-2c53c190bfc6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.220944 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.220754 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c09d668-aa2a-494f-902c-2c53c190bfc6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.220944 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.220780 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c09d668-aa2a-494f-902c-2c53c190bfc6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.220944 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.220803 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcp2g\" (UniqueName: \"kubernetes.io/projected/4c09d668-aa2a-494f-902c-2c53c190bfc6-kube-api-access-dcp2g\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.321734 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.321701 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c09d668-aa2a-494f-902c-2c53c190bfc6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.321734 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.321735 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c09d668-aa2a-494f-902c-2c53c190bfc6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.321981 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.321758 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c09d668-aa2a-494f-902c-2c53c190bfc6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.321981 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.321788 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcp2g\" (UniqueName: \"kubernetes.io/projected/4c09d668-aa2a-494f-902c-2c53c190bfc6-kube-api-access-dcp2g\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.321981 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.321821 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c09d668-aa2a-494f-902c-2c53c190bfc6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.321981 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.321866 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c09d668-aa2a-494f-902c-2c53c190bfc6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.321981 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.321913 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c09d668-aa2a-494f-902c-2c53c190bfc6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.321981 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.321939 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c09d668-aa2a-494f-902c-2c53c190bfc6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.321981 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.321978 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c09d668-aa2a-494f-902c-2c53c190bfc6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.322678 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.322446 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c09d668-aa2a-494f-902c-2c53c190bfc6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.322678 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.322528 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c09d668-aa2a-494f-902c-2c53c190bfc6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.322678 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.322611 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c09d668-aa2a-494f-902c-2c53c190bfc6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.322678 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.322627 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c09d668-aa2a-494f-902c-2c53c190bfc6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.322678 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.322645 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c09d668-aa2a-494f-902c-2c53c190bfc6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.324637 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.324580 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c09d668-aa2a-494f-902c-2c53c190bfc6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.324743 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.324727 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c09d668-aa2a-494f-902c-2c53c190bfc6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.330960 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.330937 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c09d668-aa2a-494f-902c-2c53c190bfc6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.331090 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.331074 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcp2g\" (UniqueName: \"kubernetes.io/projected/4c09d668-aa2a-494f-902c-2c53c190bfc6-kube-api-access-dcp2g\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj\" (UID: \"4c09d668-aa2a-494f-902c-2c53c190bfc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.340012 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.339994 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj"] Jul 26 16:14:21.341822 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:14:21.341797 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a037c20_6177_4eea_ab0c_beef73b93bf6.slice/crio-4c73f867d93e1c100aad4087c2a0a6e3d2a16949a1cf5a5067763590f5782534 WatchSource:0}: Error finding container 4c73f867d93e1c100aad4087c2a0a6e3d2a16949a1cf5a5067763590f5782534: Status 404 returned error can't find the container with id 4c73f867d93e1c100aad4087c2a0a6e3d2a16949a1cf5a5067763590f5782534 Jul 26 16:14:21.451695 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.451617 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:21.513060 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.511454 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9"] Jul 26 16:14:21.515208 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.515183 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.519002 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.518671 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gxk9t\"" Jul 26 16:14:21.519002 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.518746 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 26 16:14:21.519002 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.518844 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 26 16:14:21.519002 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.518955 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 26 16:14:21.527033 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.526138 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9"] Jul 26 16:14:21.553521 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.553489 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" event={"ID":"9a037c20-6177-4eea-ab0c-beef73b93bf6","Type":"ContainerStarted","Data":"4c73f867d93e1c100aad4087c2a0a6e3d2a16949a1cf5a5067763590f5782534"} Jul 26 16:14:21.604543 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.604517 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj"] Jul 26 16:14:21.607094 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:14:21.607066 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c09d668_aa2a_494f_902c_2c53c190bfc6.slice/crio-e9a08f560712df47236852d5716e32a279679e3429afda007dc9010687ae09ec WatchSource:0}: Error finding container e9a08f560712df47236852d5716e32a279679e3429afda007dc9010687ae09ec: Status 404 returned error can't find the container with id e9a08f560712df47236852d5716e32a279679e3429afda007dc9010687ae09ec Jul 26 16:14:21.626161 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.626138 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f4215e-9b81-4a1e-8c75-68edb9616df2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.626285 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.626170 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c6f4215e-9b81-4a1e-8c75-68edb9616df2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.626285 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.626263 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c6f4215e-9b81-4a1e-8c75-68edb9616df2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.626362 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.626294 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxsgd\" (UniqueName: \"kubernetes.io/projected/c6f4215e-9b81-4a1e-8c75-68edb9616df2-kube-api-access-jxsgd\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.626362 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.626321 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f4215e-9b81-4a1e-8c75-68edb9616df2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.626362 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.626340 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c6f4215e-9b81-4a1e-8c75-68edb9616df2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.626550 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.626399 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c6f4215e-9b81-4a1e-8c75-68edb9616df2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.727529 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.727440 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f4215e-9b81-4a1e-8c75-68edb9616df2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.727529 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.727485 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c6f4215e-9b81-4a1e-8c75-68edb9616df2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.727759 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.727539 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c6f4215e-9b81-4a1e-8c75-68edb9616df2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.727759 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.727573 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxsgd\" (UniqueName: \"kubernetes.io/projected/c6f4215e-9b81-4a1e-8c75-68edb9616df2-kube-api-access-jxsgd\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.727759 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.727632 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f4215e-9b81-4a1e-8c75-68edb9616df2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.727759 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.727673 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c6f4215e-9b81-4a1e-8c75-68edb9616df2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.727759 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.727717 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c6f4215e-9b81-4a1e-8c75-68edb9616df2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.728378 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.728353 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c6f4215e-9b81-4a1e-8c75-68edb9616df2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.730053 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.730002 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c6f4215e-9b81-4a1e-8c75-68edb9616df2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.730169 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.730061 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c6f4215e-9b81-4a1e-8c75-68edb9616df2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.730314 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.730295 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f4215e-9b81-4a1e-8c75-68edb9616df2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.730380 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.730328 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c6f4215e-9b81-4a1e-8c75-68edb9616df2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.736391 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.736367 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f4215e-9b81-4a1e-8c75-68edb9616df2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.736740 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.736723 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxsgd\" (UniqueName: \"kubernetes.io/projected/c6f4215e-9b81-4a1e-8c75-68edb9616df2-kube-api-access-jxsgd\") pod \"istiod-openshift-gateway-5677ccb49b-8g8p9\" (UID: \"c6f4215e-9b81-4a1e-8c75-68edb9616df2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.828408 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.828371 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:21.975868 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:21.975837 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9"] Jul 26 16:14:21.978715 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:14:21.978654 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f4215e_9b81_4a1e_8c75_68edb9616df2.slice/crio-7e3768e128326b692dbfe50c53169aa4137707f51aff9f17162896d5181eb258 WatchSource:0}: Error finding container 7e3768e128326b692dbfe50c53169aa4137707f51aff9f17162896d5181eb258: Status 404 returned error can't find the container with id 7e3768e128326b692dbfe50c53169aa4137707f51aff9f17162896d5181eb258 Jul 26 16:14:22.562171 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:22.562100 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" event={"ID":"c6f4215e-9b81-4a1e-8c75-68edb9616df2","Type":"ContainerStarted","Data":"7e3768e128326b692dbfe50c53169aa4137707f51aff9f17162896d5181eb258"} Jul 26 16:14:22.564088 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:22.564014 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" event={"ID":"4c09d668-aa2a-494f-902c-2c53c190bfc6","Type":"ContainerStarted","Data":"e9a08f560712df47236852d5716e32a279679e3429afda007dc9010687ae09ec"} Jul 26 16:14:25.498620 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:25.498575 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:14:25.499034 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:25.498650 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:14:25.499034 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:25.498680 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:14:25.499034 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:25.498723 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:14:25.499034 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:25.498797 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:14:25.505790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:25.505765 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:14:25.505892 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:25.505824 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:14:25.505892 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:25.505850 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:14:26.591837 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:26.591792 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" event={"ID":"c6f4215e-9b81-4a1e-8c75-68edb9616df2","Type":"ContainerStarted","Data":"3f17d24d6c7d9cfc0a98f2f746b3332f8a6c0968cd3c1ed490c9dfe282bd8db1"} Jul 26 16:14:26.592319 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:26.592004 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:26.593586 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:26.593547 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" event={"ID":"9a037c20-6177-4eea-ab0c-beef73b93bf6","Type":"ContainerStarted","Data":"52614a364fc845805c309fdeca15622242224363a0c5098df9e1061d12fff345"} Jul 26 16:14:26.594019 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:26.593990 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" Jul 26 16:14:26.595181 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:26.595132 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" event={"ID":"4c09d668-aa2a-494f-902c-2c53c190bfc6","Type":"ContainerStarted","Data":"404e3cdf2f395658e16a16042aa455b7e5411737aa7592ab2b3630c8ad0d893c"} Jul 26 16:14:26.613507 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:26.613466 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8g8p9" podStartSLOduration=2.095692288 podStartE2EDuration="5.613453667s" totalImagesPullingTime="3.517761379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:14:26.61175074 +0000 UTC m=+340.192837322" watchObservedRunningTime="2026-07-26 16:14:26.613453667 +0000 UTC m=+340.194540238" Jul 26 16:14:26.632358 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:26.632310 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" podStartSLOduration=1.735837619 podStartE2EDuration="5.632295785s" totalImagesPullingTime="3.896458166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:14:26.630839064 +0000 UTC m=+340.211925674" watchObservedRunningTime="2026-07-26 16:14:26.632295785 +0000 UTC m=+340.213382354" Jul 26 16:14:26.650734 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:26.650681 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" podStartSLOduration=2.496427498 podStartE2EDuration="6.650665077s" totalImagesPullingTime="4.154237579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:14:26.649334649 +0000 UTC m=+340.230421219" watchObservedRunningTime="2026-07-26 16:14:26.650665077 +0000 UTC m=+340.231751647" Jul 26 16:14:27.189309 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:27.189271 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:27.190567 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:27.190541 2563 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Jul 26 16:14:27.190697 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:27.190596 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" podUID="9a037c20-6177-4eea-ab0c-beef73b93bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Jul 26 16:14:27.451961 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:27.451874 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:27.456817 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:27.456786 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:27.600653 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:27.600619 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:27.601629 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:27.601609 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj" Jul 26 16:14:27.655259 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:27.655213 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj"] Jul 26 16:14:28.189150 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:28.189113 2563 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Jul 26 16:14:28.189400 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:28.189171 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" podUID="9a037c20-6177-4eea-ab0c-beef73b93bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Jul 26 16:14:29.189173 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:29.189132 2563 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Jul 26 16:14:29.189567 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:29.189192 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" podUID="9a037c20-6177-4eea-ab0c-beef73b93bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Jul 26 16:14:29.612056 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:29.612021 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" podUID="9a037c20-6177-4eea-ab0c-beef73b93bf6" containerName="istio-proxy" containerID="cri-o://52614a364fc845805c309fdeca15622242224363a0c5098df9e1061d12fff345" gracePeriod=30 Jul 26 16:14:34.863036 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.863005 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:34.955135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.955048 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-podinfo\") pod \"9a037c20-6177-4eea-ab0c-beef73b93bf6\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " Jul 26 16:14:34.955135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.955084 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-envoy\") pod \"9a037c20-6177-4eea-ab0c-beef73b93bf6\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " Jul 26 16:14:34.955135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.955113 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-workload-certs\") pod \"9a037c20-6177-4eea-ab0c-beef73b93bf6\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " Jul 26 16:14:34.955135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.955130 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-token\") pod \"9a037c20-6177-4eea-ab0c-beef73b93bf6\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " Jul 26 16:14:34.955408 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.955144 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s2xj\" (UniqueName: \"kubernetes.io/projected/9a037c20-6177-4eea-ab0c-beef73b93bf6-kube-api-access-9s2xj\") pod \"9a037c20-6177-4eea-ab0c-beef73b93bf6\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " Jul 26 16:14:34.955408 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.955254 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a037c20-6177-4eea-ab0c-beef73b93bf6-istiod-ca-cert\") pod \"9a037c20-6177-4eea-ab0c-beef73b93bf6\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " Jul 26 16:14:34.955408 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.955286 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-workload-socket\") pod \"9a037c20-6177-4eea-ab0c-beef73b93bf6\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " Jul 26 16:14:34.955499 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.955431 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-credential-socket\") pod \"9a037c20-6177-4eea-ab0c-beef73b93bf6\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " Jul 26 16:14:34.955533 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.955520 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-data\") pod \"9a037c20-6177-4eea-ab0c-beef73b93bf6\" (UID: \"9a037c20-6177-4eea-ab0c-beef73b93bf6\") " Jul 26 16:14:34.956700 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.956475 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9a037c20-6177-4eea-ab0c-beef73b93bf6" (UID: "9a037c20-6177-4eea-ab0c-beef73b93bf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:14:34.956700 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.956619 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-data" (OuterVolumeSpecName: "istio-data") pod "9a037c20-6177-4eea-ab0c-beef73b93bf6" (UID: "9a037c20-6177-4eea-ab0c-beef73b93bf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:14:34.956700 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.956632 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9a037c20-6177-4eea-ab0c-beef73b93bf6" (UID: "9a037c20-6177-4eea-ab0c-beef73b93bf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:14:34.956700 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.956651 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9a037c20-6177-4eea-ab0c-beef73b93bf6" (UID: "9a037c20-6177-4eea-ab0c-beef73b93bf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:14:34.956700 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.956664 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a037c20-6177-4eea-ab0c-beef73b93bf6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9a037c20-6177-4eea-ab0c-beef73b93bf6" (UID: "9a037c20-6177-4eea-ab0c-beef73b93bf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:14:34.958627 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.958602 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9a037c20-6177-4eea-ab0c-beef73b93bf6" (UID: "9a037c20-6177-4eea-ab0c-beef73b93bf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:14:34.958732 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.958623 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-token" (OuterVolumeSpecName: "istio-token") pod "9a037c20-6177-4eea-ab0c-beef73b93bf6" (UID: "9a037c20-6177-4eea-ab0c-beef73b93bf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:14:34.958732 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.958623 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a037c20-6177-4eea-ab0c-beef73b93bf6-kube-api-access-9s2xj" (OuterVolumeSpecName: "kube-api-access-9s2xj") pod "9a037c20-6177-4eea-ab0c-beef73b93bf6" (UID: "9a037c20-6177-4eea-ab0c-beef73b93bf6"). InnerVolumeSpecName "kube-api-access-9s2xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:14:34.958732 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:34.958640 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9a037c20-6177-4eea-ab0c-beef73b93bf6" (UID: "9a037c20-6177-4eea-ab0c-beef73b93bf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:14:35.057121 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:35.057082 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-credential-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:14:35.057121 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:35.057113 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-data\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:14:35.057121 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:35.057123 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-podinfo\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:14:35.057121 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:35.057131 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-envoy\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:14:35.057409 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:35.057144 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-workload-certs\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:14:35.057409 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:35.057155 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a037c20-6177-4eea-ab0c-beef73b93bf6-istio-token\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:14:35.057409 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:35.057163 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s2xj\" (UniqueName: \"kubernetes.io/projected/9a037c20-6177-4eea-ab0c-beef73b93bf6-kube-api-access-9s2xj\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:14:35.057409 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:35.057171 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a037c20-6177-4eea-ab0c-beef73b93bf6-istiod-ca-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:14:35.057409 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:35.057179 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a037c20-6177-4eea-ab0c-beef73b93bf6-workload-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:14:35.642540 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:35.642502 2563 generic.go:356] "Generic (PLEG): container finished" podID="9a037c20-6177-4eea-ab0c-beef73b93bf6" containerID="52614a364fc845805c309fdeca15622242224363a0c5098df9e1061d12fff345" exitCode=0 Jul 26 16:14:35.642738 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:35.642549 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" Jul 26 16:14:35.642738 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:35.642558 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" event={"ID":"9a037c20-6177-4eea-ab0c-beef73b93bf6","Type":"ContainerDied","Data":"52614a364fc845805c309fdeca15622242224363a0c5098df9e1061d12fff345"} Jul 26 16:14:35.642738 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:35.642613 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj" event={"ID":"9a037c20-6177-4eea-ab0c-beef73b93bf6","Type":"ContainerDied","Data":"4c73f867d93e1c100aad4087c2a0a6e3d2a16949a1cf5a5067763590f5782534"} Jul 26 16:14:35.642738 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:35.642642 2563 scope.go:117] "RemoveContainer" containerID="52614a364fc845805c309fdeca15622242224363a0c5098df9e1061d12fff345" Jul 26 16:14:35.653990 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:35.653965 2563 scope.go:117] "RemoveContainer" containerID="52614a364fc845805c309fdeca15622242224363a0c5098df9e1061d12fff345" Jul 26 16:14:35.654499 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:14:35.654474 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52614a364fc845805c309fdeca15622242224363a0c5098df9e1061d12fff345\": container with ID starting with 52614a364fc845805c309fdeca15622242224363a0c5098df9e1061d12fff345 not found: ID does not exist" containerID="52614a364fc845805c309fdeca15622242224363a0c5098df9e1061d12fff345" Jul 26 16:14:35.654572 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:35.654510 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52614a364fc845805c309fdeca15622242224363a0c5098df9e1061d12fff345"} err="failed to get container status \"52614a364fc845805c309fdeca15622242224363a0c5098df9e1061d12fff345\": rpc error: code = NotFound desc = could not find container \"52614a364fc845805c309fdeca15622242224363a0c5098df9e1061d12fff345\": container with ID starting with 52614a364fc845805c309fdeca15622242224363a0c5098df9e1061d12fff345 not found: ID does not exist" Jul 26 16:14:35.664882 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:35.664857 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj"] Jul 26 16:14:35.668636 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:35.668614 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-9tgcj"] Jul 26 16:14:36.890269 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:14:36.890239 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a037c20-6177-4eea-ab0c-beef73b93bf6" path="/var/lib/kubelet/pods/9a037c20-6177-4eea-ab0c-beef73b93bf6/volumes" Jul 26 16:15:30.438720 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.438674 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z"] Jul 26 16:15:30.439152 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.439018 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a037c20-6177-4eea-ab0c-beef73b93bf6" containerName="istio-proxy" Jul 26 16:15:30.439152 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.439029 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a037c20-6177-4eea-ab0c-beef73b93bf6" containerName="istio-proxy" Jul 26 16:15:30.439152 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.439098 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a037c20-6177-4eea-ab0c-beef73b93bf6" containerName="istio-proxy" Jul 26 16:15:30.442188 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.442170 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z" Jul 26 16:15:30.444937 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.444915 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 26 16:15:30.446147 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.446127 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5kmdp\"" Jul 26 16:15:30.446147 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.446142 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 26 16:15:30.450319 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.450286 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z"] Jul 26 16:15:30.542606 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.542574 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs9fw\" (UniqueName: \"kubernetes.io/projected/2c801712-0ec8-41d7-ba1a-6e0616e58361-kube-api-access-cs9fw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z\" (UID: \"2c801712-0ec8-41d7-ba1a-6e0616e58361\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z" Jul 26 16:15:30.542606 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.542611 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c801712-0ec8-41d7-ba1a-6e0616e58361-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z\" (UID: \"2c801712-0ec8-41d7-ba1a-6e0616e58361\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z" Jul 26 16:15:30.542851 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.542640 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c801712-0ec8-41d7-ba1a-6e0616e58361-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z\" (UID: \"2c801712-0ec8-41d7-ba1a-6e0616e58361\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z" Jul 26 16:15:30.644062 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.644024 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs9fw\" (UniqueName: \"kubernetes.io/projected/2c801712-0ec8-41d7-ba1a-6e0616e58361-kube-api-access-cs9fw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z\" (UID: \"2c801712-0ec8-41d7-ba1a-6e0616e58361\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z" Jul 26 16:15:30.644062 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.644067 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c801712-0ec8-41d7-ba1a-6e0616e58361-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z\" (UID: \"2c801712-0ec8-41d7-ba1a-6e0616e58361\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z" Jul 26 16:15:30.644304 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.644102 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c801712-0ec8-41d7-ba1a-6e0616e58361-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z\" (UID: \"2c801712-0ec8-41d7-ba1a-6e0616e58361\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z" Jul 26 16:15:30.644493 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.644473 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c801712-0ec8-41d7-ba1a-6e0616e58361-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z\" (UID: \"2c801712-0ec8-41d7-ba1a-6e0616e58361\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z" Jul 26 16:15:30.644535 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.644508 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c801712-0ec8-41d7-ba1a-6e0616e58361-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z\" (UID: \"2c801712-0ec8-41d7-ba1a-6e0616e58361\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z" Jul 26 16:15:30.654306 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.654281 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs9fw\" (UniqueName: \"kubernetes.io/projected/2c801712-0ec8-41d7-ba1a-6e0616e58361-kube-api-access-cs9fw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z\" (UID: \"2c801712-0ec8-41d7-ba1a-6e0616e58361\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z" Jul 26 16:15:30.752173 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.752079 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z" Jul 26 16:15:30.880873 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:30.880844 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z"] Jul 26 16:15:30.882649 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:15:30.882625 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c801712_0ec8_41d7_ba1a_6e0616e58361.slice/crio-ec0b876eba16cc186daf0c72610b8914e17bb20b2189274459090c67d0409315 WatchSource:0}: Error finding container ec0b876eba16cc186daf0c72610b8914e17bb20b2189274459090c67d0409315: Status 404 returned error can't find the container with id ec0b876eba16cc186daf0c72610b8914e17bb20b2189274459090c67d0409315 Jul 26 16:15:31.235505 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.235470 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp"] Jul 26 16:15:31.238991 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.238970 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp" Jul 26 16:15:31.248353 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.248328 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp"] Jul 26 16:15:31.349499 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.349459 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8369e121-2656-4b33-a7de-53e136a71530-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp\" (UID: \"8369e121-2656-4b33-a7de-53e136a71530\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp" Jul 26 16:15:31.349685 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.349504 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4cvn\" (UniqueName: \"kubernetes.io/projected/8369e121-2656-4b33-a7de-53e136a71530-kube-api-access-l4cvn\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp\" (UID: \"8369e121-2656-4b33-a7de-53e136a71530\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp" Jul 26 16:15:31.349685 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.349543 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8369e121-2656-4b33-a7de-53e136a71530-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp\" (UID: \"8369e121-2656-4b33-a7de-53e136a71530\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp" Jul 26 16:15:31.450294 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.450251 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8369e121-2656-4b33-a7de-53e136a71530-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp\" (UID: \"8369e121-2656-4b33-a7de-53e136a71530\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp" Jul 26 16:15:31.450294 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.450304 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4cvn\" (UniqueName: \"kubernetes.io/projected/8369e121-2656-4b33-a7de-53e136a71530-kube-api-access-l4cvn\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp\" (UID: \"8369e121-2656-4b33-a7de-53e136a71530\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp" Jul 26 16:15:31.450817 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.450341 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8369e121-2656-4b33-a7de-53e136a71530-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp\" (UID: \"8369e121-2656-4b33-a7de-53e136a71530\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp" Jul 26 16:15:31.450817 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.450611 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8369e121-2656-4b33-a7de-53e136a71530-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp\" (UID: \"8369e121-2656-4b33-a7de-53e136a71530\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp" Jul 26 16:15:31.450817 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.450717 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8369e121-2656-4b33-a7de-53e136a71530-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp\" (UID: \"8369e121-2656-4b33-a7de-53e136a71530\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp" Jul 26 16:15:31.459886 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.459859 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4cvn\" (UniqueName: \"kubernetes.io/projected/8369e121-2656-4b33-a7de-53e136a71530-kube-api-access-l4cvn\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp\" (UID: \"8369e121-2656-4b33-a7de-53e136a71530\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp" Jul 26 16:15:31.548646 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.548607 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp" Jul 26 16:15:31.690537 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.681493 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp"] Jul 26 16:15:31.690537 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:15:31.683679 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8369e121_2656_4b33_a7de_53e136a71530.slice/crio-1197d70de1f0ecd2fbb94a1483069dd81352dd71059f8cf8c5e9b0ee31d2e7cf WatchSource:0}: Error finding container 1197d70de1f0ecd2fbb94a1483069dd81352dd71059f8cf8c5e9b0ee31d2e7cf: Status 404 returned error can't find the container with id 1197d70de1f0ecd2fbb94a1483069dd81352dd71059f8cf8c5e9b0ee31d2e7cf Jul 26 16:15:31.836004 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.835928 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2"] Jul 26 16:15:31.839660 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.839639 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2" Jul 26 16:15:31.847259 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.847216 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2"] Jul 26 16:15:31.889353 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.889318 2563 generic.go:356] "Generic (PLEG): container finished" podID="8369e121-2656-4b33-a7de-53e136a71530" containerID="b5c718ff0de6524201a38df5f2a41a8cf1d34e6021508cd3b8a2e23504d44fbf" exitCode=0 Jul 26 16:15:31.889539 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.889389 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp" event={"ID":"8369e121-2656-4b33-a7de-53e136a71530","Type":"ContainerDied","Data":"b5c718ff0de6524201a38df5f2a41a8cf1d34e6021508cd3b8a2e23504d44fbf"} Jul 26 16:15:31.889539 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.889435 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp" event={"ID":"8369e121-2656-4b33-a7de-53e136a71530","Type":"ContainerStarted","Data":"1197d70de1f0ecd2fbb94a1483069dd81352dd71059f8cf8c5e9b0ee31d2e7cf"} Jul 26 16:15:31.890871 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.890845 2563 generic.go:356] "Generic (PLEG): container finished" podID="2c801712-0ec8-41d7-ba1a-6e0616e58361" containerID="95ac1ef2f72eeb3772103ac441c219d267d9b8a99c4dc0e752e0def50e4c04ab" exitCode=0 Jul 26 16:15:31.890957 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.890903 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z" event={"ID":"2c801712-0ec8-41d7-ba1a-6e0616e58361","Type":"ContainerDied","Data":"95ac1ef2f72eeb3772103ac441c219d267d9b8a99c4dc0e752e0def50e4c04ab"} Jul 26 16:15:31.890957 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.890933 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z" event={"ID":"2c801712-0ec8-41d7-ba1a-6e0616e58361","Type":"ContainerStarted","Data":"ec0b876eba16cc186daf0c72610b8914e17bb20b2189274459090c67d0409315"} Jul 26 16:15:31.954719 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.954685 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ae15d55-d75e-478e-beb6-044d763f1de6-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2\" (UID: \"8ae15d55-d75e-478e-beb6-044d763f1de6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2" Jul 26 16:15:31.954979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.954727 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntr7x\" (UniqueName: \"kubernetes.io/projected/8ae15d55-d75e-478e-beb6-044d763f1de6-kube-api-access-ntr7x\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2\" (UID: \"8ae15d55-d75e-478e-beb6-044d763f1de6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2" Jul 26 16:15:31.954979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:31.954810 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ae15d55-d75e-478e-beb6-044d763f1de6-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2\" (UID: \"8ae15d55-d75e-478e-beb6-044d763f1de6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2" Jul 26 16:15:32.055846 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.055810 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ae15d55-d75e-478e-beb6-044d763f1de6-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2\" (UID: \"8ae15d55-d75e-478e-beb6-044d763f1de6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2" Jul 26 16:15:32.055846 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.055855 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntr7x\" (UniqueName: \"kubernetes.io/projected/8ae15d55-d75e-478e-beb6-044d763f1de6-kube-api-access-ntr7x\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2\" (UID: \"8ae15d55-d75e-478e-beb6-044d763f1de6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2" Jul 26 16:15:32.056066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.055886 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ae15d55-d75e-478e-beb6-044d763f1de6-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2\" (UID: \"8ae15d55-d75e-478e-beb6-044d763f1de6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2" Jul 26 16:15:32.056188 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.056158 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ae15d55-d75e-478e-beb6-044d763f1de6-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2\" (UID: \"8ae15d55-d75e-478e-beb6-044d763f1de6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2" Jul 26 16:15:32.056252 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.056212 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ae15d55-d75e-478e-beb6-044d763f1de6-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2\" (UID: \"8ae15d55-d75e-478e-beb6-044d763f1de6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2" Jul 26 16:15:32.065261 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.065213 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntr7x\" (UniqueName: \"kubernetes.io/projected/8ae15d55-d75e-478e-beb6-044d763f1de6-kube-api-access-ntr7x\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2\" (UID: \"8ae15d55-d75e-478e-beb6-044d763f1de6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2" Jul 26 16:15:32.149575 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.149113 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2" Jul 26 16:15:32.241744 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.241714 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q"] Jul 26 16:15:32.246201 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.246182 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q" Jul 26 16:15:32.254202 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.254106 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q"] Jul 26 16:15:32.286624 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.286600 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2"] Jul 26 16:15:32.288402 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:15:32.288370 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae15d55_d75e_478e_beb6_044d763f1de6.slice/crio-5a3e2b8318b8f09454709c5eb5a7541c832a9618279478c64973158814ee602a WatchSource:0}: Error finding container 5a3e2b8318b8f09454709c5eb5a7541c832a9618279478c64973158814ee602a: Status 404 returned error can't find the container with id 5a3e2b8318b8f09454709c5eb5a7541c832a9618279478c64973158814ee602a Jul 26 16:15:32.358020 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.357993 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5603a10c-7a12-4635-a095-9db354be05b8-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q\" (UID: \"5603a10c-7a12-4635-a095-9db354be05b8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q" Jul 26 16:15:32.358148 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.358035 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw5lw\" (UniqueName: \"kubernetes.io/projected/5603a10c-7a12-4635-a095-9db354be05b8-kube-api-access-sw5lw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q\" (UID: \"5603a10c-7a12-4635-a095-9db354be05b8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q" Jul 26 16:15:32.358207 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.358176 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5603a10c-7a12-4635-a095-9db354be05b8-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q\" (UID: \"5603a10c-7a12-4635-a095-9db354be05b8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q" Jul 26 16:15:32.459091 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.458999 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5603a10c-7a12-4635-a095-9db354be05b8-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q\" (UID: \"5603a10c-7a12-4635-a095-9db354be05b8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q" Jul 26 16:15:32.459091 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.459050 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5603a10c-7a12-4635-a095-9db354be05b8-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q\" (UID: \"5603a10c-7a12-4635-a095-9db354be05b8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q" Jul 26 16:15:32.459594 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.459175 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw5lw\" (UniqueName: \"kubernetes.io/projected/5603a10c-7a12-4635-a095-9db354be05b8-kube-api-access-sw5lw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q\" (UID: \"5603a10c-7a12-4635-a095-9db354be05b8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q" Jul 26 16:15:32.459594 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.459402 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5603a10c-7a12-4635-a095-9db354be05b8-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q\" (UID: \"5603a10c-7a12-4635-a095-9db354be05b8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q" Jul 26 16:15:32.459594 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.459469 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5603a10c-7a12-4635-a095-9db354be05b8-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q\" (UID: \"5603a10c-7a12-4635-a095-9db354be05b8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q" Jul 26 16:15:32.468466 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.468440 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw5lw\" (UniqueName: \"kubernetes.io/projected/5603a10c-7a12-4635-a095-9db354be05b8-kube-api-access-sw5lw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q\" (UID: \"5603a10c-7a12-4635-a095-9db354be05b8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q" Jul 26 16:15:32.557982 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.557943 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q" Jul 26 16:15:32.763051 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.763019 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q"] Jul 26 16:15:32.812060 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:15:32.812027 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5603a10c_7a12_4635_a095_9db354be05b8.slice/crio-5cea221d87f7adc177ab6cc1c790d911a40357880087e572d62704979569b88f WatchSource:0}: Error finding container 5cea221d87f7adc177ab6cc1c790d911a40357880087e572d62704979569b88f: Status 404 returned error can't find the container with id 5cea221d87f7adc177ab6cc1c790d911a40357880087e572d62704979569b88f Jul 26 16:15:32.897655 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.897623 2563 generic.go:356] "Generic (PLEG): container finished" podID="8369e121-2656-4b33-a7de-53e136a71530" containerID="7251124b928f0ee012c437e5a0df6c02db9559b7d9fca1338d9ce75fc7f8664a" exitCode=0 Jul 26 16:15:32.897872 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.897673 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp" event={"ID":"8369e121-2656-4b33-a7de-53e136a71530","Type":"ContainerDied","Data":"7251124b928f0ee012c437e5a0df6c02db9559b7d9fca1338d9ce75fc7f8664a"} Jul 26 16:15:32.901768 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.901737 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q" event={"ID":"5603a10c-7a12-4635-a095-9db354be05b8","Type":"ContainerStarted","Data":"5cea221d87f7adc177ab6cc1c790d911a40357880087e572d62704979569b88f"} Jul 26 16:15:32.903387 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.903360 2563 generic.go:356] "Generic (PLEG): container finished" podID="8ae15d55-d75e-478e-beb6-044d763f1de6" containerID="d3cbadd26e2d26af9cdd1bc94649c4f8f3459c5b64baa04b90f0e10477b0ae5a" exitCode=0 Jul 26 16:15:32.903500 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.903443 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2" event={"ID":"8ae15d55-d75e-478e-beb6-044d763f1de6","Type":"ContainerDied","Data":"d3cbadd26e2d26af9cdd1bc94649c4f8f3459c5b64baa04b90f0e10477b0ae5a"} Jul 26 16:15:32.903604 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:32.903508 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2" event={"ID":"8ae15d55-d75e-478e-beb6-044d763f1de6","Type":"ContainerStarted","Data":"5a3e2b8318b8f09454709c5eb5a7541c832a9618279478c64973158814ee602a"} Jul 26 16:15:33.910394 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:33.910316 2563 generic.go:356] "Generic (PLEG): container finished" podID="8369e121-2656-4b33-a7de-53e136a71530" containerID="8d4c055946891be3d400adb7b36e3c9eb87aa5bf744c592f6b14fad4a707e636" exitCode=0 Jul 26 16:15:33.910394 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:33.910384 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp" event={"ID":"8369e121-2656-4b33-a7de-53e136a71530","Type":"ContainerDied","Data":"8d4c055946891be3d400adb7b36e3c9eb87aa5bf744c592f6b14fad4a707e636"} Jul 26 16:15:33.912119 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:33.912091 2563 generic.go:356] "Generic (PLEG): container finished" podID="2c801712-0ec8-41d7-ba1a-6e0616e58361" containerID="6959488bb041f5d560e5d01ba73cadc7fd574138a2a0ef7d73be3aca0b058263" exitCode=0 Jul 26 16:15:33.912240 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:33.912159 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z" event={"ID":"2c801712-0ec8-41d7-ba1a-6e0616e58361","Type":"ContainerDied","Data":"6959488bb041f5d560e5d01ba73cadc7fd574138a2a0ef7d73be3aca0b058263"} Jul 26 16:15:33.913790 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:33.913770 2563 generic.go:356] "Generic (PLEG): container finished" podID="5603a10c-7a12-4635-a095-9db354be05b8" containerID="6fa5ded238216f6df47130bdc964a61dbdbce561c303c9aa368b2a4a439daa52" exitCode=0 Jul 26 16:15:33.913880 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:33.913851 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q" event={"ID":"5603a10c-7a12-4635-a095-9db354be05b8","Type":"ContainerDied","Data":"6fa5ded238216f6df47130bdc964a61dbdbce561c303c9aa368b2a4a439daa52"} Jul 26 16:15:33.915690 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:33.915670 2563 generic.go:356] "Generic (PLEG): container finished" podID="8ae15d55-d75e-478e-beb6-044d763f1de6" containerID="89d936957e1d22d4ef41b6b65f0edb0fc964c24adc62cc429053cebc51408c9b" exitCode=0 Jul 26 16:15:33.915773 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:33.915693 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2" event={"ID":"8ae15d55-d75e-478e-beb6-044d763f1de6","Type":"ContainerDied","Data":"89d936957e1d22d4ef41b6b65f0edb0fc964c24adc62cc429053cebc51408c9b"} Jul 26 16:15:34.927675 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:34.927561 2563 generic.go:356] "Generic (PLEG): container finished" podID="2c801712-0ec8-41d7-ba1a-6e0616e58361" containerID="c343d2072b496577fcfea9506d95d111cfa4887d758f70e07aeebd5457701962" exitCode=0 Jul 26 16:15:34.927675 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:34.927629 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z" event={"ID":"2c801712-0ec8-41d7-ba1a-6e0616e58361","Type":"ContainerDied","Data":"c343d2072b496577fcfea9506d95d111cfa4887d758f70e07aeebd5457701962"} Jul 26 16:15:34.929328 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:34.929307 2563 generic.go:356] "Generic (PLEG): container finished" podID="5603a10c-7a12-4635-a095-9db354be05b8" containerID="33fcf7e0668a30f66bcbc70607d7bbb786afa44e304ba6b4d066435befbb2d0b" exitCode=0 Jul 26 16:15:34.929454 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:34.929392 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q" event={"ID":"5603a10c-7a12-4635-a095-9db354be05b8","Type":"ContainerDied","Data":"33fcf7e0668a30f66bcbc70607d7bbb786afa44e304ba6b4d066435befbb2d0b"} Jul 26 16:15:34.931648 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:34.931629 2563 generic.go:356] "Generic (PLEG): container finished" podID="8ae15d55-d75e-478e-beb6-044d763f1de6" containerID="f41a028957a6e7184db056d937896410de4fd37a1f7ecd86b0d59e36516a0a93" exitCode=0 Jul 26 16:15:34.931902 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:34.931671 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2" event={"ID":"8ae15d55-d75e-478e-beb6-044d763f1de6","Type":"ContainerDied","Data":"f41a028957a6e7184db056d937896410de4fd37a1f7ecd86b0d59e36516a0a93"} Jul 26 16:15:35.058646 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:35.058613 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp" Jul 26 16:15:35.182471 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:35.182387 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8369e121-2656-4b33-a7de-53e136a71530-util\") pod \"8369e121-2656-4b33-a7de-53e136a71530\" (UID: \"8369e121-2656-4b33-a7de-53e136a71530\") " Jul 26 16:15:35.182471 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:35.182439 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4cvn\" (UniqueName: \"kubernetes.io/projected/8369e121-2656-4b33-a7de-53e136a71530-kube-api-access-l4cvn\") pod \"8369e121-2656-4b33-a7de-53e136a71530\" (UID: \"8369e121-2656-4b33-a7de-53e136a71530\") " Jul 26 16:15:35.182471 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:35.182457 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8369e121-2656-4b33-a7de-53e136a71530-bundle\") pod \"8369e121-2656-4b33-a7de-53e136a71530\" (UID: \"8369e121-2656-4b33-a7de-53e136a71530\") " Jul 26 16:15:35.182988 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:35.182956 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8369e121-2656-4b33-a7de-53e136a71530-bundle" (OuterVolumeSpecName: "bundle") pod "8369e121-2656-4b33-a7de-53e136a71530" (UID: "8369e121-2656-4b33-a7de-53e136a71530"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:15:35.184626 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:35.184599 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8369e121-2656-4b33-a7de-53e136a71530-kube-api-access-l4cvn" (OuterVolumeSpecName: "kube-api-access-l4cvn") pod "8369e121-2656-4b33-a7de-53e136a71530" (UID: "8369e121-2656-4b33-a7de-53e136a71530"). InnerVolumeSpecName "kube-api-access-l4cvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:15:35.187850 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:35.187824 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8369e121-2656-4b33-a7de-53e136a71530-util" (OuterVolumeSpecName: "util") pod "8369e121-2656-4b33-a7de-53e136a71530" (UID: "8369e121-2656-4b33-a7de-53e136a71530"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:15:35.284043 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:35.284007 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4cvn\" (UniqueName: \"kubernetes.io/projected/8369e121-2656-4b33-a7de-53e136a71530-kube-api-access-l4cvn\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:15:35.284043 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:35.284039 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8369e121-2656-4b33-a7de-53e136a71530-bundle\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:15:35.284043 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:35.284048 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8369e121-2656-4b33-a7de-53e136a71530-util\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:15:35.938991 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:35.938909 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp" Jul 26 16:15:35.939408 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:35.938904 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp" event={"ID":"8369e121-2656-4b33-a7de-53e136a71530","Type":"ContainerDied","Data":"1197d70de1f0ecd2fbb94a1483069dd81352dd71059f8cf8c5e9b0ee31d2e7cf"} Jul 26 16:15:35.939408 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:35.939046 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1197d70de1f0ecd2fbb94a1483069dd81352dd71059f8cf8c5e9b0ee31d2e7cf" Jul 26 16:15:35.940992 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:35.940964 2563 generic.go:356] "Generic (PLEG): container finished" podID="5603a10c-7a12-4635-a095-9db354be05b8" containerID="b113582ae41ac93ccf40482758a4e282da92cdfc50d3b50f1e81c42a4baf1d39" exitCode=0 Jul 26 16:15:35.941136 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:35.940995 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q" event={"ID":"5603a10c-7a12-4635-a095-9db354be05b8","Type":"ContainerDied","Data":"b113582ae41ac93ccf40482758a4e282da92cdfc50d3b50f1e81c42a4baf1d39"} Jul 26 16:15:36.074847 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.074819 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z" Jul 26 16:15:36.105011 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.104981 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2" Jul 26 16:15:36.191494 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.191405 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ae15d55-d75e-478e-beb6-044d763f1de6-bundle\") pod \"8ae15d55-d75e-478e-beb6-044d763f1de6\" (UID: \"8ae15d55-d75e-478e-beb6-044d763f1de6\") " Jul 26 16:15:36.191494 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.191444 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntr7x\" (UniqueName: \"kubernetes.io/projected/8ae15d55-d75e-478e-beb6-044d763f1de6-kube-api-access-ntr7x\") pod \"8ae15d55-d75e-478e-beb6-044d763f1de6\" (UID: \"8ae15d55-d75e-478e-beb6-044d763f1de6\") " Jul 26 16:15:36.191494 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.191461 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c801712-0ec8-41d7-ba1a-6e0616e58361-util\") pod \"2c801712-0ec8-41d7-ba1a-6e0616e58361\" (UID: \"2c801712-0ec8-41d7-ba1a-6e0616e58361\") " Jul 26 16:15:36.191494 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.191482 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cs9fw\" (UniqueName: \"kubernetes.io/projected/2c801712-0ec8-41d7-ba1a-6e0616e58361-kube-api-access-cs9fw\") pod \"2c801712-0ec8-41d7-ba1a-6e0616e58361\" (UID: \"2c801712-0ec8-41d7-ba1a-6e0616e58361\") " Jul 26 16:15:36.191823 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.191607 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ae15d55-d75e-478e-beb6-044d763f1de6-util\") pod \"8ae15d55-d75e-478e-beb6-044d763f1de6\" (UID: \"8ae15d55-d75e-478e-beb6-044d763f1de6\") " Jul 26 16:15:36.191823 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.191651 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c801712-0ec8-41d7-ba1a-6e0616e58361-bundle\") pod \"2c801712-0ec8-41d7-ba1a-6e0616e58361\" (UID: \"2c801712-0ec8-41d7-ba1a-6e0616e58361\") " Jul 26 16:15:36.192072 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.192001 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae15d55-d75e-478e-beb6-044d763f1de6-bundle" (OuterVolumeSpecName: "bundle") pod "8ae15d55-d75e-478e-beb6-044d763f1de6" (UID: "8ae15d55-d75e-478e-beb6-044d763f1de6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:15:36.192261 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.192196 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c801712-0ec8-41d7-ba1a-6e0616e58361-bundle" (OuterVolumeSpecName: "bundle") pod "2c801712-0ec8-41d7-ba1a-6e0616e58361" (UID: "2c801712-0ec8-41d7-ba1a-6e0616e58361"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:15:36.193735 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.193707 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae15d55-d75e-478e-beb6-044d763f1de6-kube-api-access-ntr7x" (OuterVolumeSpecName: "kube-api-access-ntr7x") pod "8ae15d55-d75e-478e-beb6-044d763f1de6" (UID: "8ae15d55-d75e-478e-beb6-044d763f1de6"). InnerVolumeSpecName "kube-api-access-ntr7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:15:36.193735 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.193707 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c801712-0ec8-41d7-ba1a-6e0616e58361-kube-api-access-cs9fw" (OuterVolumeSpecName: "kube-api-access-cs9fw") pod "2c801712-0ec8-41d7-ba1a-6e0616e58361" (UID: "2c801712-0ec8-41d7-ba1a-6e0616e58361"). InnerVolumeSpecName "kube-api-access-cs9fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:15:36.200154 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.200129 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c801712-0ec8-41d7-ba1a-6e0616e58361-util" (OuterVolumeSpecName: "util") pod "2c801712-0ec8-41d7-ba1a-6e0616e58361" (UID: "2c801712-0ec8-41d7-ba1a-6e0616e58361"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:15:36.200591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.200575 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae15d55-d75e-478e-beb6-044d763f1de6-util" (OuterVolumeSpecName: "util") pod "8ae15d55-d75e-478e-beb6-044d763f1de6" (UID: "8ae15d55-d75e-478e-beb6-044d763f1de6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:15:36.293279 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.293215 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c801712-0ec8-41d7-ba1a-6e0616e58361-bundle\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:15:36.293279 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.293273 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ae15d55-d75e-478e-beb6-044d763f1de6-bundle\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:15:36.293279 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.293283 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntr7x\" (UniqueName: \"kubernetes.io/projected/8ae15d55-d75e-478e-beb6-044d763f1de6-kube-api-access-ntr7x\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:15:36.293501 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.293294 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c801712-0ec8-41d7-ba1a-6e0616e58361-util\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:15:36.293501 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.293303 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs9fw\" (UniqueName: \"kubernetes.io/projected/2c801712-0ec8-41d7-ba1a-6e0616e58361-kube-api-access-cs9fw\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:15:36.293501 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.293311 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ae15d55-d75e-478e-beb6-044d763f1de6-util\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:15:36.946912 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.946839 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z" Jul 26 16:15:36.947370 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.946838 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z" event={"ID":"2c801712-0ec8-41d7-ba1a-6e0616e58361","Type":"ContainerDied","Data":"ec0b876eba16cc186daf0c72610b8914e17bb20b2189274459090c67d0409315"} Jul 26 16:15:36.947370 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.946962 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec0b876eba16cc186daf0c72610b8914e17bb20b2189274459090c67d0409315" Jul 26 16:15:36.948805 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.948782 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2" Jul 26 16:15:36.948935 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.948803 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2" event={"ID":"8ae15d55-d75e-478e-beb6-044d763f1de6","Type":"ContainerDied","Data":"5a3e2b8318b8f09454709c5eb5a7541c832a9618279478c64973158814ee602a"} Jul 26 16:15:36.948935 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:36.948844 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a3e2b8318b8f09454709c5eb5a7541c832a9618279478c64973158814ee602a" Jul 26 16:15:37.081699 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:37.081673 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q" Jul 26 16:15:37.202890 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:37.202802 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5603a10c-7a12-4635-a095-9db354be05b8-bundle\") pod \"5603a10c-7a12-4635-a095-9db354be05b8\" (UID: \"5603a10c-7a12-4635-a095-9db354be05b8\") " Jul 26 16:15:37.203025 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:37.202931 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sw5lw\" (UniqueName: \"kubernetes.io/projected/5603a10c-7a12-4635-a095-9db354be05b8-kube-api-access-sw5lw\") pod \"5603a10c-7a12-4635-a095-9db354be05b8\" (UID: \"5603a10c-7a12-4635-a095-9db354be05b8\") " Jul 26 16:15:37.203025 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:37.202968 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5603a10c-7a12-4635-a095-9db354be05b8-util\") pod \"5603a10c-7a12-4635-a095-9db354be05b8\" (UID: \"5603a10c-7a12-4635-a095-9db354be05b8\") " Jul 26 16:15:37.203497 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:37.203469 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5603a10c-7a12-4635-a095-9db354be05b8-bundle" (OuterVolumeSpecName: "bundle") pod "5603a10c-7a12-4635-a095-9db354be05b8" (UID: "5603a10c-7a12-4635-a095-9db354be05b8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:15:37.205059 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:37.205035 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5603a10c-7a12-4635-a095-9db354be05b8-kube-api-access-sw5lw" (OuterVolumeSpecName: "kube-api-access-sw5lw") pod "5603a10c-7a12-4635-a095-9db354be05b8" (UID: "5603a10c-7a12-4635-a095-9db354be05b8"). InnerVolumeSpecName "kube-api-access-sw5lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:15:37.208364 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:37.208326 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5603a10c-7a12-4635-a095-9db354be05b8-util" (OuterVolumeSpecName: "util") pod "5603a10c-7a12-4635-a095-9db354be05b8" (UID: "5603a10c-7a12-4635-a095-9db354be05b8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:15:37.303954 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:37.303900 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw5lw\" (UniqueName: \"kubernetes.io/projected/5603a10c-7a12-4635-a095-9db354be05b8-kube-api-access-sw5lw\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:15:37.303954 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:37.303947 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5603a10c-7a12-4635-a095-9db354be05b8-util\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:15:37.303954 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:37.303964 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5603a10c-7a12-4635-a095-9db354be05b8-bundle\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:15:37.956666 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:37.956638 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q" Jul 26 16:15:37.957050 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:37.956670 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q" event={"ID":"5603a10c-7a12-4635-a095-9db354be05b8","Type":"ContainerDied","Data":"5cea221d87f7adc177ab6cc1c790d911a40357880087e572d62704979569b88f"} Jul 26 16:15:37.957050 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:15:37.956718 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cea221d87f7adc177ab6cc1c790d911a40357880087e572d62704979569b88f" Jul 26 16:16:10.092933 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.092856 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm"] Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093185 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c801712-0ec8-41d7-ba1a-6e0616e58361" containerName="pull" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093196 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c801712-0ec8-41d7-ba1a-6e0616e58361" containerName="pull" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093204 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ae15d55-d75e-478e-beb6-044d763f1de6" containerName="util" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093210 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ae15d55-d75e-478e-beb6-044d763f1de6" containerName="util" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093244 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ae15d55-d75e-478e-beb6-044d763f1de6" containerName="extract" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093254 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ae15d55-d75e-478e-beb6-044d763f1de6" containerName="extract" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093269 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8369e121-2656-4b33-a7de-53e136a71530" containerName="pull" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093276 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8369e121-2656-4b33-a7de-53e136a71530" containerName="pull" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093282 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5603a10c-7a12-4635-a095-9db354be05b8" containerName="util" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093287 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5603a10c-7a12-4635-a095-9db354be05b8" containerName="util" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093292 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8369e121-2656-4b33-a7de-53e136a71530" containerName="extract" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093297 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8369e121-2656-4b33-a7de-53e136a71530" containerName="extract" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093305 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5603a10c-7a12-4635-a095-9db354be05b8" containerName="extract" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093310 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5603a10c-7a12-4635-a095-9db354be05b8" containerName="extract" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093317 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8369e121-2656-4b33-a7de-53e136a71530" containerName="util" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093322 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8369e121-2656-4b33-a7de-53e136a71530" containerName="util" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093329 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ae15d55-d75e-478e-beb6-044d763f1de6" containerName="pull" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093333 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ae15d55-d75e-478e-beb6-044d763f1de6" containerName="pull" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093338 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c801712-0ec8-41d7-ba1a-6e0616e58361" containerName="extract" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093343 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c801712-0ec8-41d7-ba1a-6e0616e58361" containerName="extract" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093348 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5603a10c-7a12-4635-a095-9db354be05b8" containerName="pull" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093353 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5603a10c-7a12-4635-a095-9db354be05b8" containerName="pull" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093362 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c801712-0ec8-41d7-ba1a-6e0616e58361" containerName="util" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093381 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c801712-0ec8-41d7-ba1a-6e0616e58361" containerName="util" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093435 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ae15d55-d75e-478e-beb6-044d763f1de6" containerName="extract" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093443 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5603a10c-7a12-4635-a095-9db354be05b8" containerName="extract" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093449 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c801712-0ec8-41d7-ba1a-6e0616e58361" containerName="extract" Jul 26 16:16:10.093482 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.093456 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8369e121-2656-4b33-a7de-53e136a71530" containerName="extract" Jul 26 16:16:10.096388 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.096370 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm" Jul 26 16:16:10.099257 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.099215 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 26 16:16:10.099257 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.099248 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 26 16:16:10.099424 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.099213 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tbs87\"" Jul 26 16:16:10.110249 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.110211 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm"] Jul 26 16:16:10.178407 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.178370 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8bf98a9-0524-4311-aadb-4914bb25d796-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-r42dm\" (UID: \"e8bf98a9-0524-4311-aadb-4914bb25d796\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm" Jul 26 16:16:10.178407 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.178410 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9z6g\" (UniqueName: \"kubernetes.io/projected/e8bf98a9-0524-4311-aadb-4914bb25d796-kube-api-access-b9z6g\") pod \"kuadrant-operator-controller-manager-86878c769f-r42dm\" (UID: \"e8bf98a9-0524-4311-aadb-4914bb25d796\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm" Jul 26 16:16:10.279183 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.279144 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8bf98a9-0524-4311-aadb-4914bb25d796-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-r42dm\" (UID: \"e8bf98a9-0524-4311-aadb-4914bb25d796\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm" Jul 26 16:16:10.279183 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.279184 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9z6g\" (UniqueName: \"kubernetes.io/projected/e8bf98a9-0524-4311-aadb-4914bb25d796-kube-api-access-b9z6g\") pod \"kuadrant-operator-controller-manager-86878c769f-r42dm\" (UID: \"e8bf98a9-0524-4311-aadb-4914bb25d796\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm" Jul 26 16:16:10.279583 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.279562 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8bf98a9-0524-4311-aadb-4914bb25d796-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-r42dm\" (UID: \"e8bf98a9-0524-4311-aadb-4914bb25d796\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm" Jul 26 16:16:10.294073 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.294040 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9z6g\" (UniqueName: \"kubernetes.io/projected/e8bf98a9-0524-4311-aadb-4914bb25d796-kube-api-access-b9z6g\") pod \"kuadrant-operator-controller-manager-86878c769f-r42dm\" (UID: \"e8bf98a9-0524-4311-aadb-4914bb25d796\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm" Jul 26 16:16:10.407274 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.407163 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm" Jul 26 16:16:10.563204 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:16:10.563168 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bf98a9_0524_4311_aadb_4914bb25d796.slice/crio-348c31d5738e9c648ce9b73562149e3d6de3fbe2ec3bc3391c36f517a0334318 WatchSource:0}: Error finding container 348c31d5738e9c648ce9b73562149e3d6de3fbe2ec3bc3391c36f517a0334318: Status 404 returned error can't find the container with id 348c31d5738e9c648ce9b73562149e3d6de3fbe2ec3bc3391c36f517a0334318 Jul 26 16:16:10.564253 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:10.564216 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm"] Jul 26 16:16:11.119363 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:11.119325 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm" event={"ID":"e8bf98a9-0524-4311-aadb-4914bb25d796","Type":"ContainerStarted","Data":"348c31d5738e9c648ce9b73562149e3d6de3fbe2ec3bc3391c36f517a0334318"} Jul 26 16:16:14.516462 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.516421 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76697d598c-pclxb"] Jul 26 16:16:14.520356 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.520332 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.540668 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.540601 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76697d598c-pclxb"] Jul 26 16:16:14.619187 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.619142 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a48906d8-915f-4211-9319-5b5786f92eee-console-serving-cert\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.619187 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.619188 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a48906d8-915f-4211-9319-5b5786f92eee-trusted-ca-bundle\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.619471 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.619240 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxdrq\" (UniqueName: \"kubernetes.io/projected/a48906d8-915f-4211-9319-5b5786f92eee-kube-api-access-wxdrq\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.619471 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.619272 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a48906d8-915f-4211-9319-5b5786f92eee-console-oauth-config\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.619471 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.619360 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a48906d8-915f-4211-9319-5b5786f92eee-console-config\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.619471 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.619447 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a48906d8-915f-4211-9319-5b5786f92eee-service-ca\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.619685 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.619472 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a48906d8-915f-4211-9319-5b5786f92eee-oauth-serving-cert\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.720313 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.720270 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a48906d8-915f-4211-9319-5b5786f92eee-console-config\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.720509 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.720353 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a48906d8-915f-4211-9319-5b5786f92eee-service-ca\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.720509 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.720377 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a48906d8-915f-4211-9319-5b5786f92eee-oauth-serving-cert\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.720509 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.720457 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a48906d8-915f-4211-9319-5b5786f92eee-console-serving-cert\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.720509 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.720486 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a48906d8-915f-4211-9319-5b5786f92eee-trusted-ca-bundle\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.720733 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.720547 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxdrq\" (UniqueName: \"kubernetes.io/projected/a48906d8-915f-4211-9319-5b5786f92eee-kube-api-access-wxdrq\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.720733 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.720706 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a48906d8-915f-4211-9319-5b5786f92eee-console-oauth-config\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.721165 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.721114 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a48906d8-915f-4211-9319-5b5786f92eee-console-config\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.721393 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.721358 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a48906d8-915f-4211-9319-5b5786f92eee-oauth-serving-cert\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.721554 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.721531 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a48906d8-915f-4211-9319-5b5786f92eee-service-ca\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.721658 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.721638 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a48906d8-915f-4211-9319-5b5786f92eee-trusted-ca-bundle\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.723467 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.723441 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a48906d8-915f-4211-9319-5b5786f92eee-console-serving-cert\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.723467 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.723456 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a48906d8-915f-4211-9319-5b5786f92eee-console-oauth-config\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.729022 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.728994 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxdrq\" (UniqueName: \"kubernetes.io/projected/a48906d8-915f-4211-9319-5b5786f92eee-kube-api-access-wxdrq\") pod \"console-76697d598c-pclxb\" (UID: \"a48906d8-915f-4211-9319-5b5786f92eee\") " pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:14.837980 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:14.837939 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:15.578039 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:15.578015 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76697d598c-pclxb"] Jul 26 16:16:15.579731 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:16:15.579708 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48906d8_915f_4211_9319_5b5786f92eee.slice/crio-1c719d632ed9c617bf4459616ff643a2e7fa6b2fc91f2dcf326083d5e33ac8bb WatchSource:0}: Error finding container 1c719d632ed9c617bf4459616ff643a2e7fa6b2fc91f2dcf326083d5e33ac8bb: Status 404 returned error can't find the container with id 1c719d632ed9c617bf4459616ff643a2e7fa6b2fc91f2dcf326083d5e33ac8bb Jul 26 16:16:16.150238 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:16.150184 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm" event={"ID":"e8bf98a9-0524-4311-aadb-4914bb25d796","Type":"ContainerStarted","Data":"080cf4630bd0ced5525f9749b2b84a678f98cf55079cdc1b6c69b61da458d87e"} Jul 26 16:16:16.150433 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:16.150275 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm" Jul 26 16:16:16.151965 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:16.151938 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76697d598c-pclxb" event={"ID":"a48906d8-915f-4211-9319-5b5786f92eee","Type":"ContainerStarted","Data":"b659dff82f037d5d12eb42cb31ce35bc1ad746c4fb1e2fa8adae257dbd5e8920"} Jul 26 16:16:16.152078 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:16.151979 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76697d598c-pclxb" event={"ID":"a48906d8-915f-4211-9319-5b5786f92eee","Type":"ContainerStarted","Data":"1c719d632ed9c617bf4459616ff643a2e7fa6b2fc91f2dcf326083d5e33ac8bb"} Jul 26 16:16:16.173649 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:16.173593 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm" podStartSLOduration=1.222819944 podStartE2EDuration="6.173578744s" totalImagesPullingTime="4.9507588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:16:16.171760926 +0000 UTC m=+449.752847496" watchObservedRunningTime="2026-07-26 16:16:16.173578744 +0000 UTC m=+449.754665315" Jul 26 16:16:16.190517 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:16.190462 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76697d598c-pclxb" podStartSLOduration=2.190435679 podStartE2EDuration="2.190435679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:16:16.190144014 +0000 UTC m=+449.771230585" watchObservedRunningTime="2026-07-26 16:16:16.190435679 +0000 UTC m=+449.771522250" Jul 26 16:16:24.838522 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:24.838471 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:24.839056 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:24.838572 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:24.843160 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:24.843135 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:25.203316 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:25.203233 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76697d598c-pclxb" Jul 26 16:16:25.250341 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:25.250300 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69c9b49979-67cns"] Jul 26 16:16:27.159779 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:27.159749 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm" Jul 26 16:16:50.278239 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.278189 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-69c9b49979-67cns" podUID="81a471a5-49b8-416f-90a4-e5700cd96bdf" containerName="console" containerID="cri-o://9f049d3fdb50eb2fbb136b79ef855f2676aae737225fb35eabef1735bfa6cded" gracePeriod=15 Jul 26 16:16:50.533133 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.533072 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:16:50.629289 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.629242 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81a471a5-49b8-416f-90a4-e5700cd96bdf-console-serving-cert\") pod \"81a471a5-49b8-416f-90a4-e5700cd96bdf\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " Jul 26 16:16:50.629472 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.629320 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-console-config\") pod \"81a471a5-49b8-416f-90a4-e5700cd96bdf\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " Jul 26 16:16:50.629472 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.629343 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-trusted-ca-bundle\") pod \"81a471a5-49b8-416f-90a4-e5700cd96bdf\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " Jul 26 16:16:50.629472 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.629365 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81a471a5-49b8-416f-90a4-e5700cd96bdf-console-oauth-config\") pod \"81a471a5-49b8-416f-90a4-e5700cd96bdf\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " Jul 26 16:16:50.629472 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.629405 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brg6r\" (UniqueName: \"kubernetes.io/projected/81a471a5-49b8-416f-90a4-e5700cd96bdf-kube-api-access-brg6r\") pod \"81a471a5-49b8-416f-90a4-e5700cd96bdf\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " Jul 26 16:16:50.629685 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.629485 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-service-ca\") pod \"81a471a5-49b8-416f-90a4-e5700cd96bdf\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " Jul 26 16:16:50.629685 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.629632 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-oauth-serving-cert\") pod \"81a471a5-49b8-416f-90a4-e5700cd96bdf\" (UID: \"81a471a5-49b8-416f-90a4-e5700cd96bdf\") " Jul 26 16:16:50.629858 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.629804 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-console-config" (OuterVolumeSpecName: "console-config") pod "81a471a5-49b8-416f-90a4-e5700cd96bdf" (UID: "81a471a5-49b8-416f-90a4-e5700cd96bdf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:16:50.630053 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.629811 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "81a471a5-49b8-416f-90a4-e5700cd96bdf" (UID: "81a471a5-49b8-416f-90a4-e5700cd96bdf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:16:50.630053 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.629885 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-service-ca" (OuterVolumeSpecName: "service-ca") pod "81a471a5-49b8-416f-90a4-e5700cd96bdf" (UID: "81a471a5-49b8-416f-90a4-e5700cd96bdf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:16:50.630053 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.629943 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-service-ca\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:16:50.630053 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.629957 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-console-config\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:16:50.630053 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.630020 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-trusted-ca-bundle\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:16:50.630358 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.630132 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "81a471a5-49b8-416f-90a4-e5700cd96bdf" (UID: "81a471a5-49b8-416f-90a4-e5700cd96bdf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:16:50.631630 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.631607 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81a471a5-49b8-416f-90a4-e5700cd96bdf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "81a471a5-49b8-416f-90a4-e5700cd96bdf" (UID: "81a471a5-49b8-416f-90a4-e5700cd96bdf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:16:50.631713 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.631653 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81a471a5-49b8-416f-90a4-e5700cd96bdf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "81a471a5-49b8-416f-90a4-e5700cd96bdf" (UID: "81a471a5-49b8-416f-90a4-e5700cd96bdf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:16:50.631713 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.631674 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a471a5-49b8-416f-90a4-e5700cd96bdf-kube-api-access-brg6r" (OuterVolumeSpecName: "kube-api-access-brg6r") pod "81a471a5-49b8-416f-90a4-e5700cd96bdf" (UID: "81a471a5-49b8-416f-90a4-e5700cd96bdf"). InnerVolumeSpecName "kube-api-access-brg6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:16:50.730568 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.730531 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81a471a5-49b8-416f-90a4-e5700cd96bdf-console-oauth-config\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:16:50.730568 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.730561 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brg6r\" (UniqueName: \"kubernetes.io/projected/81a471a5-49b8-416f-90a4-e5700cd96bdf-kube-api-access-brg6r\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:16:50.730568 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.730571 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81a471a5-49b8-416f-90a4-e5700cd96bdf-oauth-serving-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:16:50.730798 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:50.730580 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81a471a5-49b8-416f-90a4-e5700cd96bdf-console-serving-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:16:51.333887 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:51.333853 2563 generic.go:356] "Generic (PLEG): container finished" podID="81a471a5-49b8-416f-90a4-e5700cd96bdf" containerID="9f049d3fdb50eb2fbb136b79ef855f2676aae737225fb35eabef1735bfa6cded" exitCode=0 Jul 26 16:16:51.334287 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:51.333899 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c9b49979-67cns" Jul 26 16:16:51.334287 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:51.333941 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c9b49979-67cns" event={"ID":"81a471a5-49b8-416f-90a4-e5700cd96bdf","Type":"ContainerDied","Data":"9f049d3fdb50eb2fbb136b79ef855f2676aae737225fb35eabef1735bfa6cded"} Jul 26 16:16:51.334287 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:51.333995 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c9b49979-67cns" event={"ID":"81a471a5-49b8-416f-90a4-e5700cd96bdf","Type":"ContainerDied","Data":"2d11e4e340b21cc90efe43a1535b08f039fb57464ac9f1a93ffb3906cd41477f"} Jul 26 16:16:51.334287 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:51.334021 2563 scope.go:117] "RemoveContainer" containerID="9f049d3fdb50eb2fbb136b79ef855f2676aae737225fb35eabef1735bfa6cded" Jul 26 16:16:51.343640 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:51.343583 2563 scope.go:117] "RemoveContainer" containerID="9f049d3fdb50eb2fbb136b79ef855f2676aae737225fb35eabef1735bfa6cded" Jul 26 16:16:51.343894 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:16:51.343874 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f049d3fdb50eb2fbb136b79ef855f2676aae737225fb35eabef1735bfa6cded\": container with ID starting with 9f049d3fdb50eb2fbb136b79ef855f2676aae737225fb35eabef1735bfa6cded not found: ID does not exist" containerID="9f049d3fdb50eb2fbb136b79ef855f2676aae737225fb35eabef1735bfa6cded" Jul 26 16:16:51.343963 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:51.343905 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f049d3fdb50eb2fbb136b79ef855f2676aae737225fb35eabef1735bfa6cded"} err="failed to get container status \"9f049d3fdb50eb2fbb136b79ef855f2676aae737225fb35eabef1735bfa6cded\": rpc error: code = NotFound desc = could not find container \"9f049d3fdb50eb2fbb136b79ef855f2676aae737225fb35eabef1735bfa6cded\": container with ID starting with 9f049d3fdb50eb2fbb136b79ef855f2676aae737225fb35eabef1735bfa6cded not found: ID does not exist" Jul 26 16:16:51.354640 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:51.354615 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69c9b49979-67cns"] Jul 26 16:16:51.357794 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:51.357769 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69c9b49979-67cns"] Jul 26 16:16:52.890005 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:16:52.889967 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81a471a5-49b8-416f-90a4-e5700cd96bdf" path="/var/lib/kubelet/pods/81a471a5-49b8-416f-90a4-e5700cd96bdf/volumes" Jul 26 16:17:29.413110 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:29.413073 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57d9cd5db-2zv7w"] Jul 26 16:17:29.413774 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:29.413440 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81a471a5-49b8-416f-90a4-e5700cd96bdf" containerName="console" Jul 26 16:17:29.413774 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:29.413455 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="81a471a5-49b8-416f-90a4-e5700cd96bdf" containerName="console" Jul 26 16:17:29.413774 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:29.413517 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="81a471a5-49b8-416f-90a4-e5700cd96bdf" containerName="console" Jul 26 16:17:29.416783 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:29.416764 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d9cd5db-2zv7w" Jul 26 16:17:29.419559 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:29.419538 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 26 16:17:29.419758 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:29.419739 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gqw2f\"" Jul 26 16:17:29.419820 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:29.419801 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 26 16:17:29.429931 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:29.429907 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57d9cd5db-2zv7w"] Jul 26 16:17:29.572204 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:29.572171 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1adbc9e1-757d-4343-a247-60f21d829c3b-webhook-cert\") pod \"maas-controller-57d9cd5db-2zv7w\" (UID: \"1adbc9e1-757d-4343-a247-60f21d829c3b\") " pod="opendatahub/maas-controller-57d9cd5db-2zv7w" Jul 26 16:17:29.572403 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:29.572254 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9hqc\" (UniqueName: \"kubernetes.io/projected/1adbc9e1-757d-4343-a247-60f21d829c3b-kube-api-access-g9hqc\") pod \"maas-controller-57d9cd5db-2zv7w\" (UID: \"1adbc9e1-757d-4343-a247-60f21d829c3b\") " pod="opendatahub/maas-controller-57d9cd5db-2zv7w" Jul 26 16:17:29.673214 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:29.673112 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9hqc\" (UniqueName: \"kubernetes.io/projected/1adbc9e1-757d-4343-a247-60f21d829c3b-kube-api-access-g9hqc\") pod \"maas-controller-57d9cd5db-2zv7w\" (UID: \"1adbc9e1-757d-4343-a247-60f21d829c3b\") " pod="opendatahub/maas-controller-57d9cd5db-2zv7w" Jul 26 16:17:29.673413 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:29.673245 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1adbc9e1-757d-4343-a247-60f21d829c3b-webhook-cert\") pod \"maas-controller-57d9cd5db-2zv7w\" (UID: \"1adbc9e1-757d-4343-a247-60f21d829c3b\") " pod="opendatahub/maas-controller-57d9cd5db-2zv7w" Jul 26 16:17:29.675722 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:29.675693 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1adbc9e1-757d-4343-a247-60f21d829c3b-webhook-cert\") pod \"maas-controller-57d9cd5db-2zv7w\" (UID: \"1adbc9e1-757d-4343-a247-60f21d829c3b\") " pod="opendatahub/maas-controller-57d9cd5db-2zv7w" Jul 26 16:17:29.682892 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:29.682859 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9hqc\" (UniqueName: \"kubernetes.io/projected/1adbc9e1-757d-4343-a247-60f21d829c3b-kube-api-access-g9hqc\") pod \"maas-controller-57d9cd5db-2zv7w\" (UID: \"1adbc9e1-757d-4343-a247-60f21d829c3b\") " pod="opendatahub/maas-controller-57d9cd5db-2zv7w" Jul 26 16:17:29.727472 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:29.727433 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d9cd5db-2zv7w" Jul 26 16:17:29.884948 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:29.884923 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57d9cd5db-2zv7w"] Jul 26 16:17:29.887237 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:17:29.887189 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbc9e1_757d_4343_a247_60f21d829c3b.slice/crio-0b5427d9d2c885cae16d19e8692058656e61853c174c92153f31f700519440d6 WatchSource:0}: Error finding container 0b5427d9d2c885cae16d19e8692058656e61853c174c92153f31f700519440d6: Status 404 returned error can't find the container with id 0b5427d9d2c885cae16d19e8692058656e61853c174c92153f31f700519440d6 Jul 26 16:17:30.529611 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:30.529572 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d9cd5db-2zv7w" event={"ID":"1adbc9e1-757d-4343-a247-60f21d829c3b","Type":"ContainerStarted","Data":"0b5427d9d2c885cae16d19e8692058656e61853c174c92153f31f700519440d6"} Jul 26 16:17:32.542847 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:32.542750 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d9cd5db-2zv7w" event={"ID":"1adbc9e1-757d-4343-a247-60f21d829c3b","Type":"ContainerStarted","Data":"941c135d6ea56b5cefe0101283024b2e2799074373e3ac125d0ebd8072d02c96"} Jul 26 16:17:32.543199 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:32.542864 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57d9cd5db-2zv7w" Jul 26 16:17:32.560677 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:32.560620 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57d9cd5db-2zv7w" podStartSLOduration=1.164055247 podStartE2EDuration="3.56060036s" totalImagesPullingTime="2.396545113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:17:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:17:32.560148312 +0000 UTC m=+526.141234881" watchObservedRunningTime="2026-07-26 16:17:32.56060036 +0000 UTC m=+526.141686931" Jul 26 16:17:43.550549 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:43.550478 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57d9cd5db-2zv7w" Jul 26 16:17:43.680419 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:43.680381 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57d9cd5db-2zv7w"] Jul 26 16:17:43.680611 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:43.680569 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-57d9cd5db-2zv7w" podUID="1adbc9e1-757d-4343-a247-60f21d829c3b" containerName="manager" containerID="cri-o://941c135d6ea56b5cefe0101283024b2e2799074373e3ac125d0ebd8072d02c96" gracePeriod=30 Jul 26 16:17:43.928202 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:43.928177 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d9cd5db-2zv7w" Jul 26 16:17:43.994976 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:43.994941 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1adbc9e1-757d-4343-a247-60f21d829c3b-webhook-cert\") pod \"1adbc9e1-757d-4343-a247-60f21d829c3b\" (UID: \"1adbc9e1-757d-4343-a247-60f21d829c3b\") " Jul 26 16:17:43.995173 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:43.995069 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9hqc\" (UniqueName: \"kubernetes.io/projected/1adbc9e1-757d-4343-a247-60f21d829c3b-kube-api-access-g9hqc\") pod \"1adbc9e1-757d-4343-a247-60f21d829c3b\" (UID: \"1adbc9e1-757d-4343-a247-60f21d829c3b\") " Jul 26 16:17:43.997085 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:43.997055 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1adbc9e1-757d-4343-a247-60f21d829c3b-kube-api-access-g9hqc" (OuterVolumeSpecName: "kube-api-access-g9hqc") pod "1adbc9e1-757d-4343-a247-60f21d829c3b" (UID: "1adbc9e1-757d-4343-a247-60f21d829c3b"). InnerVolumeSpecName "kube-api-access-g9hqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:17:43.997174 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:43.997081 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1adbc9e1-757d-4343-a247-60f21d829c3b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1adbc9e1-757d-4343-a247-60f21d829c3b" (UID: "1adbc9e1-757d-4343-a247-60f21d829c3b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:17:44.095750 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:44.095712 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9hqc\" (UniqueName: \"kubernetes.io/projected/1adbc9e1-757d-4343-a247-60f21d829c3b-kube-api-access-g9hqc\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:17:44.095750 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:44.095746 2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1adbc9e1-757d-4343-a247-60f21d829c3b-webhook-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:17:44.602208 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:44.602172 2563 generic.go:356] "Generic (PLEG): container finished" podID="1adbc9e1-757d-4343-a247-60f21d829c3b" containerID="941c135d6ea56b5cefe0101283024b2e2799074373e3ac125d0ebd8072d02c96" exitCode=0 Jul 26 16:17:44.602699 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:44.602243 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d9cd5db-2zv7w" event={"ID":"1adbc9e1-757d-4343-a247-60f21d829c3b","Type":"ContainerDied","Data":"941c135d6ea56b5cefe0101283024b2e2799074373e3ac125d0ebd8072d02c96"} Jul 26 16:17:44.602699 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:44.602252 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d9cd5db-2zv7w" Jul 26 16:17:44.602699 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:44.602313 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d9cd5db-2zv7w" event={"ID":"1adbc9e1-757d-4343-a247-60f21d829c3b","Type":"ContainerDied","Data":"0b5427d9d2c885cae16d19e8692058656e61853c174c92153f31f700519440d6"} Jul 26 16:17:44.602699 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:44.602336 2563 scope.go:117] "RemoveContainer" containerID="941c135d6ea56b5cefe0101283024b2e2799074373e3ac125d0ebd8072d02c96" Jul 26 16:17:44.611401 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:44.611383 2563 scope.go:117] "RemoveContainer" containerID="941c135d6ea56b5cefe0101283024b2e2799074373e3ac125d0ebd8072d02c96" Jul 26 16:17:44.611673 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:17:44.611654 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941c135d6ea56b5cefe0101283024b2e2799074373e3ac125d0ebd8072d02c96\": container with ID starting with 941c135d6ea56b5cefe0101283024b2e2799074373e3ac125d0ebd8072d02c96 not found: ID does not exist" containerID="941c135d6ea56b5cefe0101283024b2e2799074373e3ac125d0ebd8072d02c96" Jul 26 16:17:44.611716 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:44.611680 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941c135d6ea56b5cefe0101283024b2e2799074373e3ac125d0ebd8072d02c96"} err="failed to get container status \"941c135d6ea56b5cefe0101283024b2e2799074373e3ac125d0ebd8072d02c96\": rpc error: code = NotFound desc = could not find container \"941c135d6ea56b5cefe0101283024b2e2799074373e3ac125d0ebd8072d02c96\": container with ID starting with 941c135d6ea56b5cefe0101283024b2e2799074373e3ac125d0ebd8072d02c96 not found: ID does not exist" Jul 26 16:17:44.626844 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:44.626822 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57d9cd5db-2zv7w"] Jul 26 16:17:44.632874 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:44.632853 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57d9cd5db-2zv7w"] Jul 26 16:17:44.891037 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:17:44.890954 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1adbc9e1-757d-4343-a247-60f21d829c3b" path="/var/lib/kubelet/pods/1adbc9e1-757d-4343-a247-60f21d829c3b/volumes" Jul 26 16:18:06.267786 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.267753 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx"] Jul 26 16:18:06.268277 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.268063 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1adbc9e1-757d-4343-a247-60f21d829c3b" containerName="manager" Jul 26 16:18:06.268277 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.268073 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1adbc9e1-757d-4343-a247-60f21d829c3b" containerName="manager" Jul 26 16:18:06.268277 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.268135 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1adbc9e1-757d-4343-a247-60f21d829c3b" containerName="manager" Jul 26 16:18:06.275950 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.275928 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx" Jul 26 16:18:06.279640 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.279619 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c6ndf\"" Jul 26 16:18:06.280025 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.280007 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 16:18:06.280850 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.280828 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 26 16:18:06.280939 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.280862 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 26 16:18:06.293675 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.293651 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx"] Jul 26 16:18:06.387354 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.387321 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a4b6e5-6baf-4930-9329-26ea2abb176c-openshift-service-ca\") pod \"maas-api-545d4d4db9-5sbqx\" (UID: \"f9a4b6e5-6baf-4930-9329-26ea2abb176c\") " pod="odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx" Jul 26 16:18:06.387525 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.387385 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9a4b6e5-6baf-4930-9329-26ea2abb176c-maas-api-tls\") pod \"maas-api-545d4d4db9-5sbqx\" (UID: \"f9a4b6e5-6baf-4930-9329-26ea2abb176c\") " pod="odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx" Jul 26 16:18:06.387525 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.387439 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ttkf\" (UniqueName: \"kubernetes.io/projected/f9a4b6e5-6baf-4930-9329-26ea2abb176c-kube-api-access-9ttkf\") pod \"maas-api-545d4d4db9-5sbqx\" (UID: \"f9a4b6e5-6baf-4930-9329-26ea2abb176c\") " pod="odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx" Jul 26 16:18:06.488572 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.488534 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9a4b6e5-6baf-4930-9329-26ea2abb176c-maas-api-tls\") pod \"maas-api-545d4d4db9-5sbqx\" (UID: \"f9a4b6e5-6baf-4930-9329-26ea2abb176c\") " pod="odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx" Jul 26 16:18:06.488747 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.488595 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ttkf\" (UniqueName: \"kubernetes.io/projected/f9a4b6e5-6baf-4930-9329-26ea2abb176c-kube-api-access-9ttkf\") pod \"maas-api-545d4d4db9-5sbqx\" (UID: \"f9a4b6e5-6baf-4930-9329-26ea2abb176c\") " pod="odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx" Jul 26 16:18:06.488747 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.488713 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a4b6e5-6baf-4930-9329-26ea2abb176c-openshift-service-ca\") pod \"maas-api-545d4d4db9-5sbqx\" (UID: \"f9a4b6e5-6baf-4930-9329-26ea2abb176c\") " pod="odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx" Jul 26 16:18:06.489386 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.489351 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a4b6e5-6baf-4930-9329-26ea2abb176c-openshift-service-ca\") pod \"maas-api-545d4d4db9-5sbqx\" (UID: \"f9a4b6e5-6baf-4930-9329-26ea2abb176c\") " pod="odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx" Jul 26 16:18:06.490842 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.490818 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9a4b6e5-6baf-4930-9329-26ea2abb176c-maas-api-tls\") pod \"maas-api-545d4d4db9-5sbqx\" (UID: \"f9a4b6e5-6baf-4930-9329-26ea2abb176c\") " pod="odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx" Jul 26 16:18:06.497426 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.497400 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ttkf\" (UniqueName: \"kubernetes.io/projected/f9a4b6e5-6baf-4930-9329-26ea2abb176c-kube-api-access-9ttkf\") pod \"maas-api-545d4d4db9-5sbqx\" (UID: \"f9a4b6e5-6baf-4930-9329-26ea2abb176c\") " pod="odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx" Jul 26 16:18:06.586283 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.586247 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx" Jul 26 16:18:06.735022 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:06.734995 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx"] Jul 26 16:18:06.737007 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:18:06.736976 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a4b6e5_6baf_4930_9329_26ea2abb176c.slice/crio-35aa9119855cd616afb0cfb5a651c6d6458be2e15fe203810d081303a0317c15 WatchSource:0}: Error finding container 35aa9119855cd616afb0cfb5a651c6d6458be2e15fe203810d081303a0317c15: Status 404 returned error can't find the container with id 35aa9119855cd616afb0cfb5a651c6d6458be2e15fe203810d081303a0317c15 Jul 26 16:18:07.724780 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:07.724737 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx" event={"ID":"f9a4b6e5-6baf-4930-9329-26ea2abb176c","Type":"ContainerStarted","Data":"35aa9119855cd616afb0cfb5a651c6d6458be2e15fe203810d081303a0317c15"} Jul 26 16:18:08.733411 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:08.733372 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx" event={"ID":"f9a4b6e5-6baf-4930-9329-26ea2abb176c","Type":"ContainerStarted","Data":"ea24940d9d2728a51ae0e6d02b1cdcc021c638f1806c1cf2fafd23d13ebbaa6c"} Jul 26 16:18:08.733826 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:08.733495 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx" Jul 26 16:18:08.750771 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:08.750713 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx" podStartSLOduration=1.325275258 podStartE2EDuration="2.750697993s" totalImagesPullingTime="1.425422735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:18:08.749039464 +0000 UTC m=+562.330126034" watchObservedRunningTime="2026-07-26 16:18:08.750697993 +0000 UTC m=+562.331784563" Jul 26 16:18:14.745099 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:14.745069 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-545d4d4db9-5sbqx" Jul 26 16:18:54.140314 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.140179 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72"] Jul 26 16:18:54.145930 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.144708 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.148668 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.148640 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 26 16:18:54.148819 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.148667 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hhbrk\"" Jul 26 16:18:54.148819 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.148667 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 26 16:18:54.148819 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.148640 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 26 16:18:54.154524 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.154500 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72"] Jul 26 16:18:54.196031 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.195996 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lndps\" (UniqueName: \"kubernetes.io/projected/52f8bd5b-4290-4648-b305-4c478c96744f-kube-api-access-lndps\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vk72\" (UID: \"52f8bd5b-4290-4648-b305-4c478c96744f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.196189 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.196044 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f8bd5b-4290-4648-b305-4c478c96744f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vk72\" (UID: \"52f8bd5b-4290-4648-b305-4c478c96744f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.196189 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.196098 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f8bd5b-4290-4648-b305-4c478c96744f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vk72\" (UID: \"52f8bd5b-4290-4648-b305-4c478c96744f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.196189 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.196116 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f8bd5b-4290-4648-b305-4c478c96744f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vk72\" (UID: \"52f8bd5b-4290-4648-b305-4c478c96744f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.196189 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.196131 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f8bd5b-4290-4648-b305-4c478c96744f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vk72\" (UID: \"52f8bd5b-4290-4648-b305-4c478c96744f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.196358 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.196213 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f8bd5b-4290-4648-b305-4c478c96744f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vk72\" (UID: \"52f8bd5b-4290-4648-b305-4c478c96744f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.297320 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.297278 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f8bd5b-4290-4648-b305-4c478c96744f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vk72\" (UID: \"52f8bd5b-4290-4648-b305-4c478c96744f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.297617 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.297411 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lndps\" (UniqueName: \"kubernetes.io/projected/52f8bd5b-4290-4648-b305-4c478c96744f-kube-api-access-lndps\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vk72\" (UID: \"52f8bd5b-4290-4648-b305-4c478c96744f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.297617 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.297494 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f8bd5b-4290-4648-b305-4c478c96744f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vk72\" (UID: \"52f8bd5b-4290-4648-b305-4c478c96744f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.297617 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.297543 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f8bd5b-4290-4648-b305-4c478c96744f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vk72\" (UID: \"52f8bd5b-4290-4648-b305-4c478c96744f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.297617 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.297573 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f8bd5b-4290-4648-b305-4c478c96744f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vk72\" (UID: \"52f8bd5b-4290-4648-b305-4c478c96744f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.297617 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.297599 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f8bd5b-4290-4648-b305-4c478c96744f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vk72\" (UID: \"52f8bd5b-4290-4648-b305-4c478c96744f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.297975 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.297938 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f8bd5b-4290-4648-b305-4c478c96744f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vk72\" (UID: \"52f8bd5b-4290-4648-b305-4c478c96744f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.298074 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.298028 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f8bd5b-4290-4648-b305-4c478c96744f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vk72\" (UID: \"52f8bd5b-4290-4648-b305-4c478c96744f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.298243 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.298200 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f8bd5b-4290-4648-b305-4c478c96744f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vk72\" (UID: \"52f8bd5b-4290-4648-b305-4c478c96744f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.300203 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.300179 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f8bd5b-4290-4648-b305-4c478c96744f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vk72\" (UID: \"52f8bd5b-4290-4648-b305-4c478c96744f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.300425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.300288 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f8bd5b-4290-4648-b305-4c478c96744f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vk72\" (UID: \"52f8bd5b-4290-4648-b305-4c478c96744f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.305914 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.305887 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lndps\" (UniqueName: \"kubernetes.io/projected/52f8bd5b-4290-4648-b305-4c478c96744f-kube-api-access-lndps\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vk72\" (UID: \"52f8bd5b-4290-4648-b305-4c478c96744f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.456303 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.456186 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:18:54.589643 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.589613 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72"] Jul 26 16:18:54.591390 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:18:54.591361 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f8bd5b_4290_4648_b305_4c478c96744f.slice/crio-5a970f3797387bc64271e5efae5b897a99bd353b7178ba074ac38b53a158481a WatchSource:0}: Error finding container 5a970f3797387bc64271e5efae5b897a99bd353b7178ba074ac38b53a158481a: Status 404 returned error can't find the container with id 5a970f3797387bc64271e5efae5b897a99bd353b7178ba074ac38b53a158481a Jul 26 16:18:54.962281 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:54.962245 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" event={"ID":"52f8bd5b-4290-4648-b305-4c478c96744f","Type":"ContainerStarted","Data":"5a970f3797387bc64271e5efae5b897a99bd353b7178ba074ac38b53a158481a"} Jul 26 16:18:55.969678 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:55.969583 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" event={"ID":"52f8bd5b-4290-4648-b305-4c478c96744f","Type":"ContainerStarted","Data":"750493083ccd5e7ece8daf0e51f6467443a7cc5151f905dc62dd616df939493b"} Jul 26 16:18:55.989879 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:18:55.989826 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" podStartSLOduration=1.039794088 podStartE2EDuration="1.989810337s" totalImagesPullingTime="950.016249ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:18:55.986898108 +0000 UTC m=+609.567984687" watchObservedRunningTime="2026-07-26 16:18:55.989810337 +0000 UTC m=+609.570896936" Jul 26 16:19:04.456650 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:19:04.456605 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:19:04.457103 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:19:04.456677 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:19:04.468739 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:19:04.468713 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:19:05.024560 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:19:05.024530 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vk72" Jul 26 16:22:52.769614 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:52.769575 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5656c79bb6-gsbzr"] Jul 26 16:22:52.773007 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:52.772988 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5656c79bb6-gsbzr" Jul 26 16:22:52.775712 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:52.775684 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 26 16:22:52.777019 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:52.776994 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gqw2f\"" Jul 26 16:22:52.777145 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:52.777041 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 26 16:22:52.782373 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:52.782345 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5656c79bb6-gsbzr"] Jul 26 16:22:52.867846 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:52.867805 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sjhc\" (UniqueName: \"kubernetes.io/projected/cf5588b1-94f8-423d-8423-587e56db6cc3-kube-api-access-6sjhc\") pod \"maas-controller-5656c79bb6-gsbzr\" (UID: \"cf5588b1-94f8-423d-8423-587e56db6cc3\") " pod="opendatahub/maas-controller-5656c79bb6-gsbzr" Jul 26 16:22:52.868019 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:52.867897 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf5588b1-94f8-423d-8423-587e56db6cc3-webhook-cert\") pod \"maas-controller-5656c79bb6-gsbzr\" (UID: \"cf5588b1-94f8-423d-8423-587e56db6cc3\") " pod="opendatahub/maas-controller-5656c79bb6-gsbzr" Jul 26 16:22:52.968675 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:52.968637 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sjhc\" (UniqueName: \"kubernetes.io/projected/cf5588b1-94f8-423d-8423-587e56db6cc3-kube-api-access-6sjhc\") pod \"maas-controller-5656c79bb6-gsbzr\" (UID: \"cf5588b1-94f8-423d-8423-587e56db6cc3\") " pod="opendatahub/maas-controller-5656c79bb6-gsbzr" Jul 26 16:22:52.968840 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:52.968720 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf5588b1-94f8-423d-8423-587e56db6cc3-webhook-cert\") pod \"maas-controller-5656c79bb6-gsbzr\" (UID: \"cf5588b1-94f8-423d-8423-587e56db6cc3\") " pod="opendatahub/maas-controller-5656c79bb6-gsbzr" Jul 26 16:22:52.971201 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:52.971169 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf5588b1-94f8-423d-8423-587e56db6cc3-webhook-cert\") pod \"maas-controller-5656c79bb6-gsbzr\" (UID: \"cf5588b1-94f8-423d-8423-587e56db6cc3\") " pod="opendatahub/maas-controller-5656c79bb6-gsbzr" Jul 26 16:22:52.978656 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:52.978630 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sjhc\" (UniqueName: \"kubernetes.io/projected/cf5588b1-94f8-423d-8423-587e56db6cc3-kube-api-access-6sjhc\") pod \"maas-controller-5656c79bb6-gsbzr\" (UID: \"cf5588b1-94f8-423d-8423-587e56db6cc3\") " pod="opendatahub/maas-controller-5656c79bb6-gsbzr" Jul 26 16:22:53.084381 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:53.084346 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5656c79bb6-gsbzr" Jul 26 16:22:53.216252 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:53.216211 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5656c79bb6-gsbzr"] Jul 26 16:22:53.218090 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:22:53.218065 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5588b1_94f8_423d_8423_587e56db6cc3.slice/crio-fb7792333d653275368ca7a7d7e1264195bf1d3322a1b47fe4b8c47e025b467e WatchSource:0}: Error finding container fb7792333d653275368ca7a7d7e1264195bf1d3322a1b47fe4b8c47e025b467e: Status 404 returned error can't find the container with id fb7792333d653275368ca7a7d7e1264195bf1d3322a1b47fe4b8c47e025b467e Jul 26 16:22:53.219839 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:53.219817 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 16:22:54.146147 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:54.146108 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5656c79bb6-gsbzr" event={"ID":"cf5588b1-94f8-423d-8423-587e56db6cc3","Type":"ContainerStarted","Data":"83891394f42a8d5aa6dfad8390db6f205555b989afad31863fd0bb97b5339ebe"} Jul 26 16:22:54.146568 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:54.146160 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5656c79bb6-gsbzr" event={"ID":"cf5588b1-94f8-423d-8423-587e56db6cc3","Type":"ContainerStarted","Data":"fb7792333d653275368ca7a7d7e1264195bf1d3322a1b47fe4b8c47e025b467e"} Jul 26 16:22:54.146568 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:54.146190 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5656c79bb6-gsbzr" Jul 26 16:22:54.166583 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:22:54.166535 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5656c79bb6-gsbzr" podStartSLOduration=1.666036123 podStartE2EDuration="2.166520419s" totalImagesPullingTime="500.484296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:22:54.165257777 +0000 UTC m=+847.746344349" watchObservedRunningTime="2026-07-26 16:22:54.166520419 +0000 UTC m=+847.747606988" Jul 26 16:23:05.153757 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:05.153724 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5656c79bb6-gsbzr" Jul 26 16:23:09.788239 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:09.788184 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5656c79bb6-gsbzr"] Jul 26 16:23:09.788746 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:09.788415 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5656c79bb6-gsbzr" podUID="cf5588b1-94f8-423d-8423-587e56db6cc3" containerName="manager" containerID="cri-o://83891394f42a8d5aa6dfad8390db6f205555b989afad31863fd0bb97b5339ebe" gracePeriod=30 Jul 26 16:23:10.049616 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:10.049547 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5656c79bb6-gsbzr" Jul 26 16:23:10.109129 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:10.109096 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf5588b1-94f8-423d-8423-587e56db6cc3-webhook-cert\") pod \"cf5588b1-94f8-423d-8423-587e56db6cc3\" (UID: \"cf5588b1-94f8-423d-8423-587e56db6cc3\") " Jul 26 16:23:10.109352 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:10.109164 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sjhc\" (UniqueName: \"kubernetes.io/projected/cf5588b1-94f8-423d-8423-587e56db6cc3-kube-api-access-6sjhc\") pod \"cf5588b1-94f8-423d-8423-587e56db6cc3\" (UID: \"cf5588b1-94f8-423d-8423-587e56db6cc3\") " Jul 26 16:23:10.111272 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:10.111241 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5588b1-94f8-423d-8423-587e56db6cc3-kube-api-access-6sjhc" (OuterVolumeSpecName: "kube-api-access-6sjhc") pod "cf5588b1-94f8-423d-8423-587e56db6cc3" (UID: "cf5588b1-94f8-423d-8423-587e56db6cc3"). InnerVolumeSpecName "kube-api-access-6sjhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:23:10.111272 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:10.111261 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5588b1-94f8-423d-8423-587e56db6cc3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "cf5588b1-94f8-423d-8423-587e56db6cc3" (UID: "cf5588b1-94f8-423d-8423-587e56db6cc3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:23:10.210281 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:10.210246 2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf5588b1-94f8-423d-8423-587e56db6cc3-webhook-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:23:10.210281 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:10.210277 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sjhc\" (UniqueName: \"kubernetes.io/projected/cf5588b1-94f8-423d-8423-587e56db6cc3-kube-api-access-6sjhc\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:23:10.231996 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:10.231954 2563 generic.go:356] "Generic (PLEG): container finished" podID="cf5588b1-94f8-423d-8423-587e56db6cc3" containerID="83891394f42a8d5aa6dfad8390db6f205555b989afad31863fd0bb97b5339ebe" exitCode=0 Jul 26 16:23:10.231996 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:10.232001 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5656c79bb6-gsbzr" Jul 26 16:23:10.232186 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:10.232034 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5656c79bb6-gsbzr" event={"ID":"cf5588b1-94f8-423d-8423-587e56db6cc3","Type":"ContainerDied","Data":"83891394f42a8d5aa6dfad8390db6f205555b989afad31863fd0bb97b5339ebe"} Jul 26 16:23:10.232186 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:10.232110 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5656c79bb6-gsbzr" event={"ID":"cf5588b1-94f8-423d-8423-587e56db6cc3","Type":"ContainerDied","Data":"fb7792333d653275368ca7a7d7e1264195bf1d3322a1b47fe4b8c47e025b467e"} Jul 26 16:23:10.232186 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:10.232134 2563 scope.go:117] "RemoveContainer" containerID="83891394f42a8d5aa6dfad8390db6f205555b989afad31863fd0bb97b5339ebe" Jul 26 16:23:10.241439 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:10.241415 2563 scope.go:117] "RemoveContainer" containerID="83891394f42a8d5aa6dfad8390db6f205555b989afad31863fd0bb97b5339ebe" Jul 26 16:23:10.241706 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:23:10.241687 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83891394f42a8d5aa6dfad8390db6f205555b989afad31863fd0bb97b5339ebe\": container with ID starting with 83891394f42a8d5aa6dfad8390db6f205555b989afad31863fd0bb97b5339ebe not found: ID does not exist" containerID="83891394f42a8d5aa6dfad8390db6f205555b989afad31863fd0bb97b5339ebe" Jul 26 16:23:10.241788 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:10.241714 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83891394f42a8d5aa6dfad8390db6f205555b989afad31863fd0bb97b5339ebe"} err="failed to get container status \"83891394f42a8d5aa6dfad8390db6f205555b989afad31863fd0bb97b5339ebe\": rpc error: code = NotFound desc = could not find container \"83891394f42a8d5aa6dfad8390db6f205555b989afad31863fd0bb97b5339ebe\": container with ID starting with 83891394f42a8d5aa6dfad8390db6f205555b989afad31863fd0bb97b5339ebe not found: ID does not exist" Jul 26 16:23:10.255123 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:10.255100 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5656c79bb6-gsbzr"] Jul 26 16:23:10.262002 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:10.261976 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5656c79bb6-gsbzr"] Jul 26 16:23:10.890845 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:10.890812 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf5588b1-94f8-423d-8423-587e56db6cc3" path="/var/lib/kubelet/pods/cf5588b1-94f8-423d-8423-587e56db6cc3/volumes" Jul 26 16:23:11.035866 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:11.035827 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5656c79bb6-gprrd"] Jul 26 16:23:11.036187 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:11.036175 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf5588b1-94f8-423d-8423-587e56db6cc3" containerName="manager" Jul 26 16:23:11.036255 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:11.036189 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf5588b1-94f8-423d-8423-587e56db6cc3" containerName="manager" Jul 26 16:23:11.036294 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:11.036270 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf5588b1-94f8-423d-8423-587e56db6cc3" containerName="manager" Jul 26 16:23:11.040576 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:11.040557 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5656c79bb6-gprrd" Jul 26 16:23:11.044577 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:11.044551 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 26 16:23:11.044680 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:11.044564 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 26 16:23:11.044752 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:11.044558 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gqw2f\"" Jul 26 16:23:11.048030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:11.048005 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5656c79bb6-gprrd"] Jul 26 16:23:11.117583 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:11.117546 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db312ba1-76de-4f39-b952-d32b58c8306c-webhook-cert\") pod \"maas-controller-5656c79bb6-gprrd\" (UID: \"db312ba1-76de-4f39-b952-d32b58c8306c\") " pod="opendatahub/maas-controller-5656c79bb6-gprrd" Jul 26 16:23:11.117583 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:11.117587 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtgnf\" (UniqueName: \"kubernetes.io/projected/db312ba1-76de-4f39-b952-d32b58c8306c-kube-api-access-dtgnf\") pod \"maas-controller-5656c79bb6-gprrd\" (UID: \"db312ba1-76de-4f39-b952-d32b58c8306c\") " pod="opendatahub/maas-controller-5656c79bb6-gprrd" Jul 26 16:23:11.218830 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:11.218745 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db312ba1-76de-4f39-b952-d32b58c8306c-webhook-cert\") pod \"maas-controller-5656c79bb6-gprrd\" (UID: \"db312ba1-76de-4f39-b952-d32b58c8306c\") " pod="opendatahub/maas-controller-5656c79bb6-gprrd" Jul 26 16:23:11.218830 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:11.218783 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtgnf\" (UniqueName: \"kubernetes.io/projected/db312ba1-76de-4f39-b952-d32b58c8306c-kube-api-access-dtgnf\") pod \"maas-controller-5656c79bb6-gprrd\" (UID: \"db312ba1-76de-4f39-b952-d32b58c8306c\") " pod="opendatahub/maas-controller-5656c79bb6-gprrd" Jul 26 16:23:11.221123 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:11.221091 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db312ba1-76de-4f39-b952-d32b58c8306c-webhook-cert\") pod \"maas-controller-5656c79bb6-gprrd\" (UID: \"db312ba1-76de-4f39-b952-d32b58c8306c\") " pod="opendatahub/maas-controller-5656c79bb6-gprrd" Jul 26 16:23:11.227069 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:11.227041 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtgnf\" (UniqueName: \"kubernetes.io/projected/db312ba1-76de-4f39-b952-d32b58c8306c-kube-api-access-dtgnf\") pod \"maas-controller-5656c79bb6-gprrd\" (UID: \"db312ba1-76de-4f39-b952-d32b58c8306c\") " pod="opendatahub/maas-controller-5656c79bb6-gprrd" Jul 26 16:23:11.353151 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:11.353117 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5656c79bb6-gprrd" Jul 26 16:23:11.486605 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:11.486571 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5656c79bb6-gprrd"] Jul 26 16:23:11.488805 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:23:11.488773 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb312ba1_76de_4f39_b952_d32b58c8306c.slice/crio-3e64cdcf5ba193edaf7e3b7d7871c5951610d2fd4b61381ae584c1a309c6c097 WatchSource:0}: Error finding container 3e64cdcf5ba193edaf7e3b7d7871c5951610d2fd4b61381ae584c1a309c6c097: Status 404 returned error can't find the container with id 3e64cdcf5ba193edaf7e3b7d7871c5951610d2fd4b61381ae584c1a309c6c097 Jul 26 16:23:12.245255 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:12.245152 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5656c79bb6-gprrd" event={"ID":"db312ba1-76de-4f39-b952-d32b58c8306c","Type":"ContainerStarted","Data":"b003796168681040201f6e83844d6a81169078cb4085ac1efbef984de6e64654"} Jul 26 16:23:12.245255 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:12.245211 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5656c79bb6-gprrd" event={"ID":"db312ba1-76de-4f39-b952-d32b58c8306c","Type":"ContainerStarted","Data":"3e64cdcf5ba193edaf7e3b7d7871c5951610d2fd4b61381ae584c1a309c6c097"} Jul 26 16:23:12.245748 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:12.245269 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5656c79bb6-gprrd" Jul 26 16:23:12.262563 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:12.262502 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5656c79bb6-gprrd" podStartSLOduration=0.957733282 podStartE2EDuration="1.262483658s" totalImagesPullingTime="304.750376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:23:12.261625605 +0000 UTC m=+865.842712175" watchObservedRunningTime="2026-07-26 16:23:12.262483658 +0000 UTC m=+865.843570230" Jul 26 16:23:23.251804 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:23:23.251766 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5656c79bb6-gprrd" Jul 26 16:27:45.152259 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:27:45.152200 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm"] Jul 26 16:27:45.152830 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:27:45.152485 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm" podUID="e8bf98a9-0524-4311-aadb-4914bb25d796" containerName="manager" containerID="cri-o://080cf4630bd0ced5525f9749b2b84a678f98cf55079cdc1b6c69b61da458d87e" gracePeriod=10 Jul 26 16:27:48.672648 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:27:48.672601 2563 generic.go:356] "Generic (PLEG): container finished" podID="e8bf98a9-0524-4311-aadb-4914bb25d796" containerID="080cf4630bd0ced5525f9749b2b84a678f98cf55079cdc1b6c69b61da458d87e" exitCode=0 Jul 26 16:27:48.673088 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:27:48.672679 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm" event={"ID":"e8bf98a9-0524-4311-aadb-4914bb25d796","Type":"ContainerDied","Data":"080cf4630bd0ced5525f9749b2b84a678f98cf55079cdc1b6c69b61da458d87e"} Jul 26 16:27:48.695724 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:27:48.695701 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm" Jul 26 16:27:48.787709 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:27:48.787672 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9z6g\" (UniqueName: \"kubernetes.io/projected/e8bf98a9-0524-4311-aadb-4914bb25d796-kube-api-access-b9z6g\") pod \"e8bf98a9-0524-4311-aadb-4914bb25d796\" (UID: \"e8bf98a9-0524-4311-aadb-4914bb25d796\") " Jul 26 16:27:48.787870 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:27:48.787733 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8bf98a9-0524-4311-aadb-4914bb25d796-extensions-socket-volume\") pod \"e8bf98a9-0524-4311-aadb-4914bb25d796\" (UID: \"e8bf98a9-0524-4311-aadb-4914bb25d796\") " Jul 26 16:27:48.788101 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:27:48.788079 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bf98a9-0524-4311-aadb-4914bb25d796-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e8bf98a9-0524-4311-aadb-4914bb25d796" (UID: "e8bf98a9-0524-4311-aadb-4914bb25d796"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:27:48.789740 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:27:48.789716 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bf98a9-0524-4311-aadb-4914bb25d796-kube-api-access-b9z6g" (OuterVolumeSpecName: "kube-api-access-b9z6g") pod "e8bf98a9-0524-4311-aadb-4914bb25d796" (UID: "e8bf98a9-0524-4311-aadb-4914bb25d796"). InnerVolumeSpecName "kube-api-access-b9z6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:27:48.888479 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:27:48.888391 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8bf98a9-0524-4311-aadb-4914bb25d796-extensions-socket-volume\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:27:48.888479 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:27:48.888415 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9z6g\" (UniqueName: \"kubernetes.io/projected/e8bf98a9-0524-4311-aadb-4914bb25d796-kube-api-access-b9z6g\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:27:49.678828 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:27:49.678796 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm" Jul 26 16:27:49.678828 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:27:49.678811 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm" event={"ID":"e8bf98a9-0524-4311-aadb-4914bb25d796","Type":"ContainerDied","Data":"348c31d5738e9c648ce9b73562149e3d6de3fbe2ec3bc3391c36f517a0334318"} Jul 26 16:27:49.679412 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:27:49.678872 2563 scope.go:117] "RemoveContainer" containerID="080cf4630bd0ced5525f9749b2b84a678f98cf55079cdc1b6c69b61da458d87e" Jul 26 16:27:49.700114 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:27:49.700085 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm"] Jul 26 16:27:49.705982 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:27:49.705958 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r42dm"] Jul 26 16:27:50.890550 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:27:50.890513 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8bf98a9-0524-4311-aadb-4914bb25d796" path="/var/lib/kubelet/pods/e8bf98a9-0524-4311-aadb-4914bb25d796/volumes" Jul 26 16:28:46.295403 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.295366 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g2g6l"] Jul 26 16:28:46.295848 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.295712 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8bf98a9-0524-4311-aadb-4914bb25d796" containerName="manager" Jul 26 16:28:46.295848 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.295723 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8bf98a9-0524-4311-aadb-4914bb25d796" containerName="manager" Jul 26 16:28:46.295848 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.295784 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8bf98a9-0524-4311-aadb-4914bb25d796" containerName="manager" Jul 26 16:28:46.298672 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.298656 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g2g6l" Jul 26 16:28:46.301416 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.301392 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 26 16:28:46.301540 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.301472 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tbs87\"" Jul 26 16:28:46.301540 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.301479 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 26 16:28:46.311256 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.311236 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g2g6l"] Jul 26 16:28:46.420602 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.420567 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0124524d-8ad5-4512-963a-af28513c8116-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g2g6l\" (UID: \"0124524d-8ad5-4512-963a-af28513c8116\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g2g6l" Jul 26 16:28:46.420773 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.420618 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fvnr\" (UniqueName: \"kubernetes.io/projected/0124524d-8ad5-4512-963a-af28513c8116-kube-api-access-7fvnr\") pod \"kuadrant-operator-controller-manager-86878c769f-g2g6l\" (UID: \"0124524d-8ad5-4512-963a-af28513c8116\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g2g6l" Jul 26 16:28:46.521528 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.521479 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0124524d-8ad5-4512-963a-af28513c8116-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g2g6l\" (UID: \"0124524d-8ad5-4512-963a-af28513c8116\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g2g6l" Jul 26 16:28:46.521717 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.521562 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fvnr\" (UniqueName: \"kubernetes.io/projected/0124524d-8ad5-4512-963a-af28513c8116-kube-api-access-7fvnr\") pod \"kuadrant-operator-controller-manager-86878c769f-g2g6l\" (UID: \"0124524d-8ad5-4512-963a-af28513c8116\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g2g6l" Jul 26 16:28:46.521876 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.521851 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0124524d-8ad5-4512-963a-af28513c8116-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g2g6l\" (UID: \"0124524d-8ad5-4512-963a-af28513c8116\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g2g6l" Jul 26 16:28:46.535663 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.535634 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fvnr\" (UniqueName: \"kubernetes.io/projected/0124524d-8ad5-4512-963a-af28513c8116-kube-api-access-7fvnr\") pod \"kuadrant-operator-controller-manager-86878c769f-g2g6l\" (UID: \"0124524d-8ad5-4512-963a-af28513c8116\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g2g6l" Jul 26 16:28:46.609075 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.609000 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g2g6l" Jul 26 16:28:46.747758 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.747726 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g2g6l"] Jul 26 16:28:46.748878 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:28:46.748856 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0124524d_8ad5_4512_963a_af28513c8116.slice/crio-a02ea909f6e45796102db956755e83254c1167a90e37c6388b7e44cc7490e3f0 WatchSource:0}: Error finding container a02ea909f6e45796102db956755e83254c1167a90e37c6388b7e44cc7490e3f0: Status 404 returned error can't find the container with id a02ea909f6e45796102db956755e83254c1167a90e37c6388b7e44cc7490e3f0 Jul 26 16:28:46.974635 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.974602 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g2g6l" event={"ID":"0124524d-8ad5-4512-963a-af28513c8116","Type":"ContainerStarted","Data":"89f5d36f3f3c493516e5b616b527d8f28109598b5fcfa3ba70e1778e1e42fb28"} Jul 26 16:28:46.974789 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.974652 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g2g6l" event={"ID":"0124524d-8ad5-4512-963a-af28513c8116","Type":"ContainerStarted","Data":"a02ea909f6e45796102db956755e83254c1167a90e37c6388b7e44cc7490e3f0"} Jul 26 16:28:46.974789 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.974697 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g2g6l" Jul 26 16:28:46.996016 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:46.995967 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g2g6l" podStartSLOduration=0.995951976 podStartE2EDuration="995.951976ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:28:46.9938473 +0000 UTC m=+1200.574933871" watchObservedRunningTime="2026-07-26 16:28:46.995951976 +0000 UTC m=+1200.577038546" Jul 26 16:28:57.982988 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:28:57.982955 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g2g6l" Jul 26 16:31:28.745917 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.745834 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5"] Jul 26 16:31:28.749127 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.749109 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:28.751745 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.751711 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-aad57859-kserve-self-signed-certs\"" Jul 26 16:31:28.760341 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.760313 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5"] Jul 26 16:31:28.860869 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.860830 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f730c7-1931-4106-b4f2-69f15a512c38-tls-certs\") pod \"e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:28.861054 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.860880 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p86k7\" (UniqueName: \"kubernetes.io/projected/e8f730c7-1931-4106-b4f2-69f15a512c38-kube-api-access-p86k7\") pod \"e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:28.861054 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.861015 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-tmp-dir\") pod \"e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:28.861188 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.861056 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-home\") pod \"e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:28.861188 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.861124 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-model-cache\") pod \"e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:28.861188 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.861180 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-dshm\") pod \"e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:28.962683 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.962644 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-model-cache\") pod \"e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:28.962885 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.962704 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-dshm\") pod \"e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:28.962955 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.962888 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f730c7-1931-4106-b4f2-69f15a512c38-tls-certs\") pod \"e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:28.962955 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.962933 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p86k7\" (UniqueName: \"kubernetes.io/projected/e8f730c7-1931-4106-b4f2-69f15a512c38-kube-api-access-p86k7\") pod \"e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:28.963084 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.963049 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-tmp-dir\") pod \"e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:28.963146 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.963100 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-home\") pod \"e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:28.963212 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.963166 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-model-cache\") pod \"e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:28.963451 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.963430 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-home\") pod \"e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:28.963770 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.963739 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-tmp-dir\") pod \"e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:28.965312 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.965289 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-dshm\") pod \"e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:28.965560 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.965542 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f730c7-1931-4106-b4f2-69f15a512c38-tls-certs\") pod \"e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:28.971468 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:28.971437 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p86k7\" (UniqueName: \"kubernetes.io/projected/e8f730c7-1931-4106-b4f2-69f15a512c38-kube-api-access-p86k7\") pod \"e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:29.060764 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:29.060732 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:29.193570 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:29.193540 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5"] Jul 26 16:31:29.195131 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:31:29.195105 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8f730c7_1931_4106_b4f2_69f15a512c38.slice/crio-47ffed46f397dbfcaf8d8091963d6d77ba276c353438306fcc497e9facf323d2 WatchSource:0}: Error finding container 47ffed46f397dbfcaf8d8091963d6d77ba276c353438306fcc497e9facf323d2: Status 404 returned error can't find the container with id 47ffed46f397dbfcaf8d8091963d6d77ba276c353438306fcc497e9facf323d2 Jul 26 16:31:29.816026 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:29.815989 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" event={"ID":"e8f730c7-1931-4106-b4f2-69f15a512c38","Type":"ContainerStarted","Data":"959df24b2b619738ec284050c8a1aa35460d31646ef6bd3150a5ba29820dc650"} Jul 26 16:31:29.816026 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:29.816034 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" event={"ID":"e8f730c7-1931-4106-b4f2-69f15a512c38","Type":"ContainerStarted","Data":"47ffed46f397dbfcaf8d8091963d6d77ba276c353438306fcc497e9facf323d2"} Jul 26 16:31:29.834533 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:29.834483 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" podStartSLOduration=1.834464 podStartE2EDuration="1.834464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:31:29.834329202 +0000 UTC m=+1363.415415789" watchObservedRunningTime="2026-07-26 16:31:29.834464 +0000 UTC m=+1363.415550569" Jul 26 16:31:39.061821 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:39.061776 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:39.062246 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:39.061845 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:39.074452 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:39.074426 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:31:39.876749 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:31:39.876712 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:32:02.504548 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:02.504508 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5"] Jul 26 16:32:17.524366 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.524322 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" podUID="e8f730c7-1931-4106-b4f2-69f15a512c38" containerName="main" containerID="cri-o://959df24b2b619738ec284050c8a1aa35460d31646ef6bd3150a5ba29820dc650" gracePeriod=45 Jul 26 16:32:17.781111 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.781083 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:32:17.884656 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.884615 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p86k7\" (UniqueName: \"kubernetes.io/projected/e8f730c7-1931-4106-b4f2-69f15a512c38-kube-api-access-p86k7\") pod \"e8f730c7-1931-4106-b4f2-69f15a512c38\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " Jul 26 16:32:17.884849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.884664 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-home\") pod \"e8f730c7-1931-4106-b4f2-69f15a512c38\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " Jul 26 16:32:17.884849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.884694 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-tmp-dir\") pod \"e8f730c7-1931-4106-b4f2-69f15a512c38\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " Jul 26 16:32:17.884849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.884727 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-dshm\") pod \"e8f730c7-1931-4106-b4f2-69f15a512c38\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " Jul 26 16:32:17.884849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.884777 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-model-cache\") pod \"e8f730c7-1931-4106-b4f2-69f15a512c38\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " Jul 26 16:32:17.884849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.884819 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f730c7-1931-4106-b4f2-69f15a512c38-tls-certs\") pod \"e8f730c7-1931-4106-b4f2-69f15a512c38\" (UID: \"e8f730c7-1931-4106-b4f2-69f15a512c38\") " Jul 26 16:32:17.885505 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.885007 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-home" (OuterVolumeSpecName: "home") pod "e8f730c7-1931-4106-b4f2-69f15a512c38" (UID: "e8f730c7-1931-4106-b4f2-69f15a512c38"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:32:17.885505 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.885109 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8f730c7-1931-4106-b4f2-69f15a512c38" (UID: "e8f730c7-1931-4106-b4f2-69f15a512c38"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:32:17.885505 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.885150 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8f730c7-1931-4106-b4f2-69f15a512c38" (UID: "e8f730c7-1931-4106-b4f2-69f15a512c38"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:32:17.885505 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.885182 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-home\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:32:17.887249 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.887188 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8f730c7-1931-4106-b4f2-69f15a512c38-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8f730c7-1931-4106-b4f2-69f15a512c38" (UID: "e8f730c7-1931-4106-b4f2-69f15a512c38"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:32:17.887359 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.887281 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-dshm" (OuterVolumeSpecName: "dshm") pod "e8f730c7-1931-4106-b4f2-69f15a512c38" (UID: "e8f730c7-1931-4106-b4f2-69f15a512c38"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:32:17.887359 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.887322 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f730c7-1931-4106-b4f2-69f15a512c38-kube-api-access-p86k7" (OuterVolumeSpecName: "kube-api-access-p86k7") pod "e8f730c7-1931-4106-b4f2-69f15a512c38" (UID: "e8f730c7-1931-4106-b4f2-69f15a512c38"). InnerVolumeSpecName "kube-api-access-p86k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:32:17.986040 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.986008 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-dshm\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:32:17.986040 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.986034 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-model-cache\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:32:17.986040 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.986044 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f730c7-1931-4106-b4f2-69f15a512c38-tls-certs\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:32:17.986305 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.986053 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p86k7\" (UniqueName: \"kubernetes.io/projected/e8f730c7-1931-4106-b4f2-69f15a512c38-kube-api-access-p86k7\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:32:17.986305 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:17.986063 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f730c7-1931-4106-b4f2-69f15a512c38-tmp-dir\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:32:18.069337 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:18.069213 2563 generic.go:356] "Generic (PLEG): container finished" podID="e8f730c7-1931-4106-b4f2-69f15a512c38" containerID="959df24b2b619738ec284050c8a1aa35460d31646ef6bd3150a5ba29820dc650" exitCode=0 Jul 26 16:32:18.069337 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:18.069257 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" event={"ID":"e8f730c7-1931-4106-b4f2-69f15a512c38","Type":"ContainerDied","Data":"959df24b2b619738ec284050c8a1aa35460d31646ef6bd3150a5ba29820dc650"} Jul 26 16:32:18.069337 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:18.069275 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" Jul 26 16:32:18.069337 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:18.069306 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5" event={"ID":"e8f730c7-1931-4106-b4f2-69f15a512c38","Type":"ContainerDied","Data":"47ffed46f397dbfcaf8d8091963d6d77ba276c353438306fcc497e9facf323d2"} Jul 26 16:32:18.069337 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:18.069331 2563 scope.go:117] "RemoveContainer" containerID="959df24b2b619738ec284050c8a1aa35460d31646ef6bd3150a5ba29820dc650" Jul 26 16:32:18.079164 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:18.079147 2563 scope.go:117] "RemoveContainer" containerID="959df24b2b619738ec284050c8a1aa35460d31646ef6bd3150a5ba29820dc650" Jul 26 16:32:18.079417 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:32:18.079398 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"959df24b2b619738ec284050c8a1aa35460d31646ef6bd3150a5ba29820dc650\": container with ID starting with 959df24b2b619738ec284050c8a1aa35460d31646ef6bd3150a5ba29820dc650 not found: ID does not exist" containerID="959df24b2b619738ec284050c8a1aa35460d31646ef6bd3150a5ba29820dc650" Jul 26 16:32:18.079468 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:18.079425 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959df24b2b619738ec284050c8a1aa35460d31646ef6bd3150a5ba29820dc650"} err="failed to get container status \"959df24b2b619738ec284050c8a1aa35460d31646ef6bd3150a5ba29820dc650\": rpc error: code = NotFound desc = could not find container \"959df24b2b619738ec284050c8a1aa35460d31646ef6bd3150a5ba29820dc650\": container with ID starting with 959df24b2b619738ec284050c8a1aa35460d31646ef6bd3150a5ba29820dc650 not found: ID does not exist" Jul 26 16:32:18.092369 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:18.092347 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5"] Jul 26 16:32:18.096531 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:18.096501 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-aad57859-kserve-68bd69755-lj6w5"] Jul 26 16:32:18.890457 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:32:18.890422 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8f730c7-1931-4106-b4f2-69f15a512c38" path="/var/lib/kubelet/pods/e8f730c7-1931-4106-b4f2-69f15a512c38/volumes" Jul 26 16:35:45.854480 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:45.854403 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp"] Jul 26 16:35:45.855549 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:45.855519 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8f730c7-1931-4106-b4f2-69f15a512c38" containerName="main" Jul 26 16:35:45.855549 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:45.855549 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8f730c7-1931-4106-b4f2-69f15a512c38" containerName="main" Jul 26 16:35:45.855793 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:45.855772 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8f730c7-1931-4106-b4f2-69f15a512c38" containerName="main" Jul 26 16:35:45.862009 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:45.860362 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:45.865030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:45.865009 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c6b781f9-openshift-default-dockercfg-v9rfw\"" Jul 26 16:35:45.880541 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:45.880511 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp"] Jul 26 16:35:45.950528 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:45.950494 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-credential-socket\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:45.950837 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:45.950554 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-envoy\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:45.950837 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:45.950590 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-workload-certs\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:45.950837 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:45.950611 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-data\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:45.950837 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:45.950668 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-token\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:45.950837 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:45.950702 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klbxn\" (UniqueName: \"kubernetes.io/projected/c185a8c7-d142-4be0-9e47-e051555ebdec-kube-api-access-klbxn\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:45.950837 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:45.950764 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-podinfo\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:45.951189 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:45.951167 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-workload-socket\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:45.951273 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:45.951213 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c185a8c7-d142-4be0-9e47-e051555ebdec-istiod-ca-cert\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.052154 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.052118 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-podinfo\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.052425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.052178 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-workload-socket\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.052425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.052205 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c185a8c7-d142-4be0-9e47-e051555ebdec-istiod-ca-cert\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.052425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.052370 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-credential-socket\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.052626 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.052432 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-envoy\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.052626 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.052464 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-workload-certs\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.052626 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.052487 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-data\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.052626 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.052543 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-token\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.052626 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.052585 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klbxn\" (UniqueName: \"kubernetes.io/projected/c185a8c7-d142-4be0-9e47-e051555ebdec-kube-api-access-klbxn\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.052893 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.052722 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-workload-socket\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.052893 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.052752 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-credential-socket\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.053007 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.052909 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-data\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.053060 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.053044 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-workload-certs\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.053297 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.053279 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c185a8c7-d142-4be0-9e47-e051555ebdec-istiod-ca-cert\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.054638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.054617 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-envoy\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.054744 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.054725 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-podinfo\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.062208 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.062169 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klbxn\" (UniqueName: \"kubernetes.io/projected/c185a8c7-d142-4be0-9e47-e051555ebdec-kube-api-access-klbxn\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.062750 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.062646 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-token\") pod \"e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.173077 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.172985 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:46.317055 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.317027 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp"] Jul 26 16:35:46.318944 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:35:46.318913 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc185a8c7_d142_4be0_9e47_e051555ebdec.slice/crio-648f4eee647e86466e0dc7b78760a1bedb73a0994d4e75913550bb031a97115e WatchSource:0}: Error finding container 648f4eee647e86466e0dc7b78760a1bedb73a0994d4e75913550bb031a97115e: Status 404 returned error can't find the container with id 648f4eee647e86466e0dc7b78760a1bedb73a0994d4e75913550bb031a97115e Jul 26 16:35:46.320958 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.320927 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:35:46.321049 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.321004 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:35:46.321114 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:46.321045 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:35:47.143478 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:47.143440 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" event={"ID":"c185a8c7-d142-4be0-9e47-e051555ebdec","Type":"ContainerStarted","Data":"89a54302c7c13d8a67bfc1d510050c841c169f86179d81f3334f717fcec08c27"} Jul 26 16:35:47.143906 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:47.143498 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" event={"ID":"c185a8c7-d142-4be0-9e47-e051555ebdec","Type":"ContainerStarted","Data":"648f4eee647e86466e0dc7b78760a1bedb73a0994d4e75913550bb031a97115e"} Jul 26 16:35:47.170028 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:47.169971 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" podStartSLOduration=2.169956554 podStartE2EDuration="2.169956554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:35:47.16863776 +0000 UTC m=+1620.749724330" watchObservedRunningTime="2026-07-26 16:35:47.169956554 +0000 UTC m=+1620.751043124" Jul 26 16:35:47.173787 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:47.173761 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:47.178574 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:47.178552 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:48.148932 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:48.148898 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:48.149971 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:48.149954 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:35:48.772356 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:48.772309 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h"] Jul 26 16:35:48.780132 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:48.779802 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" Jul 26 16:35:48.782932 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:48.782907 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c6b781f9-serving-cert\"" Jul 26 16:35:48.790177 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:48.790152 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h"] Jul 26 16:35:48.882204 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:48.882166 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhj59\" (UniqueName: \"kubernetes.io/projected/814aab71-b6ac-465d-82c9-b4221ad57075-kube-api-access-vhj59\") pod \"maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h\" (UID: \"814aab71-b6ac-465d-82c9-b4221ad57075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" Jul 26 16:35:48.882404 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:48.882278 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/814aab71-b6ac-465d-82c9-b4221ad57075-openshift-service-ca\") pod \"maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h\" (UID: \"814aab71-b6ac-465d-82c9-b4221ad57075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" Jul 26 16:35:48.882404 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:48.882326 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/814aab71-b6ac-465d-82c9-b4221ad57075-maas-api-tls\") pod \"maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h\" (UID: \"814aab71-b6ac-465d-82c9-b4221ad57075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" Jul 26 16:35:48.982836 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:48.982790 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/814aab71-b6ac-465d-82c9-b4221ad57075-openshift-service-ca\") pod \"maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h\" (UID: \"814aab71-b6ac-465d-82c9-b4221ad57075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" Jul 26 16:35:48.983047 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:48.982867 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/814aab71-b6ac-465d-82c9-b4221ad57075-maas-api-tls\") pod \"maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h\" (UID: \"814aab71-b6ac-465d-82c9-b4221ad57075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" Jul 26 16:35:48.983047 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:48.982912 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhj59\" (UniqueName: \"kubernetes.io/projected/814aab71-b6ac-465d-82c9-b4221ad57075-kube-api-access-vhj59\") pod \"maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h\" (UID: \"814aab71-b6ac-465d-82c9-b4221ad57075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" Jul 26 16:35:48.983512 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:48.983489 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/814aab71-b6ac-465d-82c9-b4221ad57075-openshift-service-ca\") pod \"maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h\" (UID: \"814aab71-b6ac-465d-82c9-b4221ad57075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" Jul 26 16:35:48.985266 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:48.985249 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/814aab71-b6ac-465d-82c9-b4221ad57075-maas-api-tls\") pod \"maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h\" (UID: \"814aab71-b6ac-465d-82c9-b4221ad57075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" Jul 26 16:35:48.993036 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:48.993011 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhj59\" (UniqueName: \"kubernetes.io/projected/814aab71-b6ac-465d-82c9-b4221ad57075-kube-api-access-vhj59\") pod \"maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h\" (UID: \"814aab71-b6ac-465d-82c9-b4221ad57075\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" Jul 26 16:35:49.092177 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:49.092136 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" Jul 26 16:35:49.229669 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:49.229617 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h"] Jul 26 16:35:49.231719 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:35:49.231690 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod814aab71_b6ac_465d_82c9_b4221ad57075.slice/crio-2d70926f584ae31344904ebb022cf837b8aaf3b57fa303b1adcecffc001b9278 WatchSource:0}: Error finding container 2d70926f584ae31344904ebb022cf837b8aaf3b57fa303b1adcecffc001b9278: Status 404 returned error can't find the container with id 2d70926f584ae31344904ebb022cf837b8aaf3b57fa303b1adcecffc001b9278 Jul 26 16:35:49.233540 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:49.233520 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 16:35:50.162605 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:50.162556 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" event={"ID":"814aab71-b6ac-465d-82c9-b4221ad57075","Type":"ContainerStarted","Data":"068c45e527e78a777f5d1213c9a2383d7c61b98265066abe490b37a13eb1b645"} Jul 26 16:35:50.162823 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:50.162618 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" event={"ID":"814aab71-b6ac-465d-82c9-b4221ad57075","Type":"ContainerStarted","Data":"2d70926f584ae31344904ebb022cf837b8aaf3b57fa303b1adcecffc001b9278"} Jul 26 16:35:50.162823 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:50.162647 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" Jul 26 16:35:50.213812 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:50.213752 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" podStartSLOduration=1.782170456 podStartE2EDuration="2.213737974s" totalImagesPullingTime="431.567518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:35:50.211373435 +0000 UTC m=+1623.792460004" watchObservedRunningTime="2026-07-26 16:35:50.213737974 +0000 UTC m=+1623.794824545" Jul 26 16:35:56.179568 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:35:56.179480 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" Jul 26 16:36:00.223786 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:00.223749 2563 generic.go:356] "Generic (PLEG): container finished" podID="c185a8c7-d142-4be0-9e47-e051555ebdec" containerID="89a54302c7c13d8a67bfc1d510050c841c169f86179d81f3334f717fcec08c27" exitCode=0 Jul 26 16:36:00.224206 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:00.223819 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" event={"ID":"c185a8c7-d142-4be0-9e47-e051555ebdec","Type":"ContainerDied","Data":"89a54302c7c13d8a67bfc1d510050c841c169f86179d81f3334f717fcec08c27"} Jul 26 16:36:00.224206 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:00.224198 2563 scope.go:117] "RemoveContainer" containerID="89a54302c7c13d8a67bfc1d510050c841c169f86179d81f3334f717fcec08c27" Jul 26 16:36:00.225174 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:00.225147 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:36:00.225295 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:00.225243 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:36:00.225375 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:00.225303 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:36:01.231104 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:01.231069 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" event={"ID":"c185a8c7-d142-4be0-9e47-e051555ebdec","Type":"ContainerStarted","Data":"ce10231965508208e7dea8897ed2557178ce3a013335c9cf8e7966e80c909acb"} Jul 26 16:36:02.173612 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:02.173574 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:36:02.174567 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:02.174536 2563 patch_prober.go:28] interesting pod/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 26 16:36:02.175050 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:02.174735 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" podUID="c185a8c7-d142-4be0-9e47-e051555ebdec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 26 16:36:03.149020 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:03.148987 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:36:03.174377 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:03.174337 2563 patch_prober.go:28] interesting pod/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 26 16:36:03.175132 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:03.174685 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" podUID="c185a8c7-d142-4be0-9e47-e051555ebdec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 26 16:36:04.174119 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:04.174075 2563 patch_prober.go:28] interesting pod/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 26 16:36:04.174670 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:04.174152 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" podUID="c185a8c7-d142-4be0-9e47-e051555ebdec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 26 16:36:05.177382 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:05.177342 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:36:05.178471 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:05.178450 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:36:24.570962 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:24.570929 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h"] Jul 26 16:36:24.571569 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:24.571145 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" podUID="814aab71-b6ac-465d-82c9-b4221ad57075" containerName="maas-api" containerID="cri-o://068c45e527e78a777f5d1213c9a2383d7c61b98265066abe490b37a13eb1b645" gracePeriod=30 Jul 26 16:36:24.830997 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:24.830927 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" Jul 26 16:36:25.019235 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:25.019199 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/814aab71-b6ac-465d-82c9-b4221ad57075-maas-api-tls\") pod \"814aab71-b6ac-465d-82c9-b4221ad57075\" (UID: \"814aab71-b6ac-465d-82c9-b4221ad57075\") " Jul 26 16:36:25.019432 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:25.019319 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhj59\" (UniqueName: \"kubernetes.io/projected/814aab71-b6ac-465d-82c9-b4221ad57075-kube-api-access-vhj59\") pod \"814aab71-b6ac-465d-82c9-b4221ad57075\" (UID: \"814aab71-b6ac-465d-82c9-b4221ad57075\") " Jul 26 16:36:25.019432 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:25.019412 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/814aab71-b6ac-465d-82c9-b4221ad57075-openshift-service-ca\") pod \"814aab71-b6ac-465d-82c9-b4221ad57075\" (UID: \"814aab71-b6ac-465d-82c9-b4221ad57075\") " Jul 26 16:36:25.019833 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:25.019795 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/814aab71-b6ac-465d-82c9-b4221ad57075-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "814aab71-b6ac-465d-82c9-b4221ad57075" (UID: "814aab71-b6ac-465d-82c9-b4221ad57075"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:36:25.021546 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:25.021515 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/814aab71-b6ac-465d-82c9-b4221ad57075-kube-api-access-vhj59" (OuterVolumeSpecName: "kube-api-access-vhj59") pod "814aab71-b6ac-465d-82c9-b4221ad57075" (UID: "814aab71-b6ac-465d-82c9-b4221ad57075"). InnerVolumeSpecName "kube-api-access-vhj59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:36:25.021733 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:25.021713 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/814aab71-b6ac-465d-82c9-b4221ad57075-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "814aab71-b6ac-465d-82c9-b4221ad57075" (UID: "814aab71-b6ac-465d-82c9-b4221ad57075"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:36:25.120477 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:25.120387 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhj59\" (UniqueName: \"kubernetes.io/projected/814aab71-b6ac-465d-82c9-b4221ad57075-kube-api-access-vhj59\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:36:25.120477 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:25.120419 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/814aab71-b6ac-465d-82c9-b4221ad57075-openshift-service-ca\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:36:25.120477 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:25.120429 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/814aab71-b6ac-465d-82c9-b4221ad57075-maas-api-tls\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:36:25.369733 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:25.369689 2563 generic.go:356] "Generic (PLEG): container finished" podID="814aab71-b6ac-465d-82c9-b4221ad57075" containerID="068c45e527e78a777f5d1213c9a2383d7c61b98265066abe490b37a13eb1b645" exitCode=0 Jul 26 16:36:25.369936 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:25.369746 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" Jul 26 16:36:25.369936 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:25.369778 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" event={"ID":"814aab71-b6ac-465d-82c9-b4221ad57075","Type":"ContainerDied","Data":"068c45e527e78a777f5d1213c9a2383d7c61b98265066abe490b37a13eb1b645"} Jul 26 16:36:25.369936 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:25.369863 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h" event={"ID":"814aab71-b6ac-465d-82c9-b4221ad57075","Type":"ContainerDied","Data":"2d70926f584ae31344904ebb022cf837b8aaf3b57fa303b1adcecffc001b9278"} Jul 26 16:36:25.369936 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:25.369896 2563 scope.go:117] "RemoveContainer" containerID="068c45e527e78a777f5d1213c9a2383d7c61b98265066abe490b37a13eb1b645" Jul 26 16:36:25.380160 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:25.380141 2563 scope.go:117] "RemoveContainer" containerID="068c45e527e78a777f5d1213c9a2383d7c61b98265066abe490b37a13eb1b645" Jul 26 16:36:25.380489 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:36:25.380469 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"068c45e527e78a777f5d1213c9a2383d7c61b98265066abe490b37a13eb1b645\": container with ID starting with 068c45e527e78a777f5d1213c9a2383d7c61b98265066abe490b37a13eb1b645 not found: ID does not exist" containerID="068c45e527e78a777f5d1213c9a2383d7c61b98265066abe490b37a13eb1b645" Jul 26 16:36:25.380555 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:25.380500 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"068c45e527e78a777f5d1213c9a2383d7c61b98265066abe490b37a13eb1b645"} err="failed to get container status \"068c45e527e78a777f5d1213c9a2383d7c61b98265066abe490b37a13eb1b645\": rpc error: code = NotFound desc = could not find container \"068c45e527e78a777f5d1213c9a2383d7c61b98265066abe490b37a13eb1b645\": container with ID starting with 068c45e527e78a777f5d1213c9a2383d7c61b98265066abe490b37a13eb1b645 not found: ID does not exist" Jul 26 16:36:25.395073 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:25.395043 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h"] Jul 26 16:36:25.402146 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:25.402117 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6b781f9-68646d7bdb-68c5h"] Jul 26 16:36:26.890553 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:26.890518 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="814aab71-b6ac-465d-82c9-b4221ad57075" path="/var/lib/kubelet/pods/814aab71-b6ac-465d-82c9-b4221ad57075/volumes" Jul 26 16:36:29.319062 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:29.319030 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp"] Jul 26 16:36:29.319530 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:29.319272 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" podUID="c185a8c7-d142-4be0-9e47-e051555ebdec" containerName="istio-proxy" containerID="cri-o://ce10231965508208e7dea8897ed2557178ce3a013335c9cf8e7966e80c909acb" gracePeriod=30 Jul 26 16:36:30.435496 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.435461 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh"] Jul 26 16:36:30.435957 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.435829 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="814aab71-b6ac-465d-82c9-b4221ad57075" containerName="maas-api" Jul 26 16:36:30.435957 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.435840 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="814aab71-b6ac-465d-82c9-b4221ad57075" containerName="maas-api" Jul 26 16:36:30.435957 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.435903 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="814aab71-b6ac-465d-82c9-b4221ad57075" containerName="maas-api" Jul 26 16:36:30.440352 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.440325 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.443356 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.443335 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-6f5d066a-gw-openshift-default-dockercfg-v2bp5\"" Jul 26 16:36:30.452243 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.452190 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh"] Jul 26 16:36:30.466480 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.466452 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-envoy\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.466633 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.466485 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-token\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.466633 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.466521 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-data\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.466633 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.466600 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-credential-socket\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.466769 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.466645 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs2nt\" (UniqueName: \"kubernetes.io/projected/5712ffe2-1838-4e97-ba55-2a914d43f040-kube-api-access-qs2nt\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.466769 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.466671 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-podinfo\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.466769 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.466729 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-workload-certs\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.466769 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.466763 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-workload-socket\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.466896 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.466778 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5712ffe2-1838-4e97-ba55-2a914d43f040-istiod-ca-cert\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.568151 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.568120 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-data\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.568361 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.568203 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-credential-socket\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.568361 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.568256 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs2nt\" (UniqueName: \"kubernetes.io/projected/5712ffe2-1838-4e97-ba55-2a914d43f040-kube-api-access-qs2nt\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.568361 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.568287 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-podinfo\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.568361 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.568334 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-workload-certs\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.568595 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.568384 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-workload-socket\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.568595 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.568410 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5712ffe2-1838-4e97-ba55-2a914d43f040-istiod-ca-cert\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.568595 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.568472 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-envoy\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.568595 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.568497 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-token\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.568813 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.568664 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-credential-socket\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.568919 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.568895 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-workload-socket\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.569011 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.568913 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-data\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.569011 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.568969 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-workload-certs\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.569306 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.569284 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5712ffe2-1838-4e97-ba55-2a914d43f040-istiod-ca-cert\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.570742 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.570724 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-envoy\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.571048 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.571027 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-podinfo\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.588832 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.588804 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-token\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.590577 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.590560 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs2nt\" (UniqueName: \"kubernetes.io/projected/5712ffe2-1838-4e97-ba55-2a914d43f040-kube-api-access-qs2nt\") pod \"e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.751492 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.751403 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:30.894505 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.894477 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh"] Jul 26 16:36:30.895972 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:36:30.895945 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5712ffe2_1838_4e97_ba55_2a914d43f040.slice/crio-b5f820b36dced4bfeb9befd6fbdfbe5ad4eb5cf626b0e4bef1510fee61c86144 WatchSource:0}: Error finding container b5f820b36dced4bfeb9befd6fbdfbe5ad4eb5cf626b0e4bef1510fee61c86144: Status 404 returned error can't find the container with id b5f820b36dced4bfeb9befd6fbdfbe5ad4eb5cf626b0e4bef1510fee61c86144 Jul 26 16:36:30.898003 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.897974 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:36:30.898071 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.898034 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:36:30.898114 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:30.898074 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:36:31.404799 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:31.404761 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" event={"ID":"5712ffe2-1838-4e97-ba55-2a914d43f040","Type":"ContainerStarted","Data":"879687ead7a00af9a3d2271462aaa3ed448eda1ddb516e52ef9c42a2a4d048db"} Jul 26 16:36:31.404997 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:31.404841 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" event={"ID":"5712ffe2-1838-4e97-ba55-2a914d43f040","Type":"ContainerStarted","Data":"b5f820b36dced4bfeb9befd6fbdfbe5ad4eb5cf626b0e4bef1510fee61c86144"} Jul 26 16:36:31.446032 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:31.445974 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" podStartSLOduration=1.445959009 podStartE2EDuration="1.445959009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:36:31.444820015 +0000 UTC m=+1665.025906585" watchObservedRunningTime="2026-07-26 16:36:31.445959009 +0000 UTC m=+1665.027045578" Jul 26 16:36:31.752390 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:31.752302 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:31.753668 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:31.753644 2563 patch_prober.go:28] interesting pod/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 26 16:36:31.753734 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:31.753704 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" podUID="5712ffe2-1838-4e97-ba55-2a914d43f040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 26 16:36:32.755717 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:32.755686 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:33.416753 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:33.416722 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:33.417822 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:33.417804 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:34.577885 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.577821 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:36:34.603336 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.603305 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-workload-socket\") pod \"c185a8c7-d142-4be0-9e47-e051555ebdec\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " Jul 26 16:36:34.603500 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.603351 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-workload-certs\") pod \"c185a8c7-d142-4be0-9e47-e051555ebdec\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " Jul 26 16:36:34.603500 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.603368 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c185a8c7-d142-4be0-9e47-e051555ebdec-istiod-ca-cert\") pod \"c185a8c7-d142-4be0-9e47-e051555ebdec\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " Jul 26 16:36:34.603500 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.603393 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-credential-socket\") pod \"c185a8c7-d142-4be0-9e47-e051555ebdec\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " Jul 26 16:36:34.603500 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.603439 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-podinfo\") pod \"c185a8c7-d142-4be0-9e47-e051555ebdec\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " Jul 26 16:36:34.603500 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.603455 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klbxn\" (UniqueName: \"kubernetes.io/projected/c185a8c7-d142-4be0-9e47-e051555ebdec-kube-api-access-klbxn\") pod \"c185a8c7-d142-4be0-9e47-e051555ebdec\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " Jul 26 16:36:34.603500 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.603481 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-envoy\") pod \"c185a8c7-d142-4be0-9e47-e051555ebdec\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " Jul 26 16:36:34.603831 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.603510 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-data\") pod \"c185a8c7-d142-4be0-9e47-e051555ebdec\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " Jul 26 16:36:34.603831 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.603534 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-token\") pod \"c185a8c7-d142-4be0-9e47-e051555ebdec\" (UID: \"c185a8c7-d142-4be0-9e47-e051555ebdec\") " Jul 26 16:36:34.603831 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.603636 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c185a8c7-d142-4be0-9e47-e051555ebdec" (UID: "c185a8c7-d142-4be0-9e47-e051555ebdec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:36:34.603831 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.603624 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c185a8c7-d142-4be0-9e47-e051555ebdec" (UID: "c185a8c7-d142-4be0-9e47-e051555ebdec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:36:34.603831 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.603753 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c185a8c7-d142-4be0-9e47-e051555ebdec" (UID: "c185a8c7-d142-4be0-9e47-e051555ebdec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:36:34.603831 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.603773 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c185a8c7-d142-4be0-9e47-e051555ebdec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c185a8c7-d142-4be0-9e47-e051555ebdec" (UID: "c185a8c7-d142-4be0-9e47-e051555ebdec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:36:34.604148 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.603895 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-workload-certs\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:36:34.604148 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.603914 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c185a8c7-d142-4be0-9e47-e051555ebdec-istiod-ca-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:36:34.604148 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.603928 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-credential-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:36:34.604148 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.603944 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-workload-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:36:34.604518 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.604487 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-data" (OuterVolumeSpecName: "istio-data") pod "c185a8c7-d142-4be0-9e47-e051555ebdec" (UID: "c185a8c7-d142-4be0-9e47-e051555ebdec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:36:34.606338 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.606079 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c185a8c7-d142-4be0-9e47-e051555ebdec" (UID: "c185a8c7-d142-4be0-9e47-e051555ebdec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:36:34.606338 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.606107 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-token" (OuterVolumeSpecName: "istio-token") pod "c185a8c7-d142-4be0-9e47-e051555ebdec" (UID: "c185a8c7-d142-4be0-9e47-e051555ebdec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:36:34.607072 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.606462 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c185a8c7-d142-4be0-9e47-e051555ebdec" (UID: "c185a8c7-d142-4be0-9e47-e051555ebdec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:36:34.607072 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.606502 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c185a8c7-d142-4be0-9e47-e051555ebdec-kube-api-access-klbxn" (OuterVolumeSpecName: "kube-api-access-klbxn") pod "c185a8c7-d142-4be0-9e47-e051555ebdec" (UID: "c185a8c7-d142-4be0-9e47-e051555ebdec"). InnerVolumeSpecName "kube-api-access-klbxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:36:34.705312 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.705277 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-podinfo\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:36:34.705312 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.705305 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klbxn\" (UniqueName: \"kubernetes.io/projected/c185a8c7-d142-4be0-9e47-e051555ebdec-kube-api-access-klbxn\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:36:34.705312 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.705317 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-envoy\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:36:34.705544 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.705325 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-data\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:36:34.705544 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:34.705333 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c185a8c7-d142-4be0-9e47-e051555ebdec-istio-token\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:36:35.430246 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:35.430191 2563 generic.go:356] "Generic (PLEG): container finished" podID="c185a8c7-d142-4be0-9e47-e051555ebdec" containerID="ce10231965508208e7dea8897ed2557178ce3a013335c9cf8e7966e80c909acb" exitCode=0 Jul 26 16:36:35.430418 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:35.430254 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" Jul 26 16:36:35.430418 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:35.430288 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" event={"ID":"c185a8c7-d142-4be0-9e47-e051555ebdec","Type":"ContainerDied","Data":"ce10231965508208e7dea8897ed2557178ce3a013335c9cf8e7966e80c909acb"} Jul 26 16:36:35.430418 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:35.430346 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp" event={"ID":"c185a8c7-d142-4be0-9e47-e051555ebdec","Type":"ContainerDied","Data":"648f4eee647e86466e0dc7b78760a1bedb73a0994d4e75913550bb031a97115e"} Jul 26 16:36:35.430418 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:35.430371 2563 scope.go:117] "RemoveContainer" containerID="ce10231965508208e7dea8897ed2557178ce3a013335c9cf8e7966e80c909acb" Jul 26 16:36:35.439194 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:35.439174 2563 scope.go:117] "RemoveContainer" containerID="89a54302c7c13d8a67bfc1d510050c841c169f86179d81f3334f717fcec08c27" Jul 26 16:36:35.448099 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:35.447908 2563 scope.go:117] "RemoveContainer" containerID="ce10231965508208e7dea8897ed2557178ce3a013335c9cf8e7966e80c909acb" Jul 26 16:36:35.448187 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:36:35.448168 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce10231965508208e7dea8897ed2557178ce3a013335c9cf8e7966e80c909acb\": container with ID starting with ce10231965508208e7dea8897ed2557178ce3a013335c9cf8e7966e80c909acb not found: ID does not exist" containerID="ce10231965508208e7dea8897ed2557178ce3a013335c9cf8e7966e80c909acb" Jul 26 16:36:35.448246 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:35.448197 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce10231965508208e7dea8897ed2557178ce3a013335c9cf8e7966e80c909acb"} err="failed to get container status \"ce10231965508208e7dea8897ed2557178ce3a013335c9cf8e7966e80c909acb\": rpc error: code = NotFound desc = could not find container \"ce10231965508208e7dea8897ed2557178ce3a013335c9cf8e7966e80c909acb\": container with ID starting with ce10231965508208e7dea8897ed2557178ce3a013335c9cf8e7966e80c909acb not found: ID does not exist" Jul 26 16:36:35.448246 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:35.448215 2563 scope.go:117] "RemoveContainer" containerID="89a54302c7c13d8a67bfc1d510050c841c169f86179d81f3334f717fcec08c27" Jul 26 16:36:35.448501 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:36:35.448483 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a54302c7c13d8a67bfc1d510050c841c169f86179d81f3334f717fcec08c27\": container with ID starting with 89a54302c7c13d8a67bfc1d510050c841c169f86179d81f3334f717fcec08c27 not found: ID does not exist" containerID="89a54302c7c13d8a67bfc1d510050c841c169f86179d81f3334f717fcec08c27" Jul 26 16:36:35.448546 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:35.448509 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a54302c7c13d8a67bfc1d510050c841c169f86179d81f3334f717fcec08c27"} err="failed to get container status \"89a54302c7c13d8a67bfc1d510050c841c169f86179d81f3334f717fcec08c27\": rpc error: code = NotFound desc = could not find container \"89a54302c7c13d8a67bfc1d510050c841c169f86179d81f3334f717fcec08c27\": container with ID starting with 89a54302c7c13d8a67bfc1d510050c841c169f86179d81f3334f717fcec08c27 not found: ID does not exist" Jul 26 16:36:35.449944 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:35.449923 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp"] Jul 26 16:36:35.453324 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:35.453300 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c6b781f9-openshift-default-8c9d4f88-l9xhp"] Jul 26 16:36:36.890154 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:36.890120 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c185a8c7-d142-4be0-9e47-e051555ebdec" path="/var/lib/kubelet/pods/c185a8c7-d142-4be0-9e47-e051555ebdec/volumes" Jul 26 16:36:47.503641 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:47.503545 2563 generic.go:356] "Generic (PLEG): container finished" podID="5712ffe2-1838-4e97-ba55-2a914d43f040" containerID="879687ead7a00af9a3d2271462aaa3ed448eda1ddb516e52ef9c42a2a4d048db" exitCode=0 Jul 26 16:36:47.503641 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:47.503621 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" event={"ID":"5712ffe2-1838-4e97-ba55-2a914d43f040","Type":"ContainerDied","Data":"879687ead7a00af9a3d2271462aaa3ed448eda1ddb516e52ef9c42a2a4d048db"} Jul 26 16:36:47.504089 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:47.504039 2563 scope.go:117] "RemoveContainer" containerID="879687ead7a00af9a3d2271462aaa3ed448eda1ddb516e52ef9c42a2a4d048db" Jul 26 16:36:47.505110 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:47.505079 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:36:47.505242 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:47.505145 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:36:47.505242 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:47.505175 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:36:48.417362 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:48.417315 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:48.510683 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:48.510645 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" event={"ID":"5712ffe2-1838-4e97-ba55-2a914d43f040","Type":"ContainerStarted","Data":"e25add81e7201a7f8655af29890dc73dccea497d28c15e85d48f6b363d909f73"} Jul 26 16:36:48.751905 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:48.751821 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:48.753038 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:48.753009 2563 patch_prober.go:28] interesting pod/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 26 16:36:48.753192 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:48.753065 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" podUID="5712ffe2-1838-4e97-ba55-2a914d43f040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 26 16:36:49.751833 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:49.751791 2563 patch_prober.go:28] interesting pod/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 26 16:36:49.752244 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:49.751850 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" podUID="5712ffe2-1838-4e97-ba55-2a914d43f040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 26 16:36:50.752700 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:50.752658 2563 patch_prober.go:28] interesting pod/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 26 16:36:50.753164 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:50.752719 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" podUID="5712ffe2-1838-4e97-ba55-2a914d43f040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 26 16:36:51.751950 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:51.751912 2563 patch_prober.go:28] interesting pod/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 26 16:36:51.752128 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:51.751969 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" podUID="5712ffe2-1838-4e97-ba55-2a914d43f040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 26 16:36:52.755926 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:52.755888 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:52.756321 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:52.756177 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:36:52.756779 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:36:52.756761 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:37:03.054450 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:03.054407 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6f5d066a-lp7cp"] Jul 26 16:37:03.054945 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:03.054923 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c185a8c7-d142-4be0-9e47-e051555ebdec" containerName="istio-proxy" Jul 26 16:37:03.055021 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:03.054949 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c185a8c7-d142-4be0-9e47-e051555ebdec" containerName="istio-proxy" Jul 26 16:37:03.055096 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:03.055082 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c185a8c7-d142-4be0-9e47-e051555ebdec" containerName="istio-proxy" Jul 26 16:37:03.058637 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:03.058614 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6f5d066a-lp7cp" Jul 26 16:37:03.062046 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:03.062027 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2xlcn\"" Jul 26 16:37:03.070816 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:03.070786 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6f5d066a-lp7cp"] Jul 26 16:37:03.084105 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:03.084070 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79c529d-ed3e-432a-96d5-544fbdd722ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6f5d066a-lp7cp\" (UID: \"a79c529d-ed3e-432a-96d5-544fbdd722ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6f5d066a-lp7cp" Jul 26 16:37:03.184682 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:03.184634 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79c529d-ed3e-432a-96d5-544fbdd722ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6f5d066a-lp7cp\" (UID: \"a79c529d-ed3e-432a-96d5-544fbdd722ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6f5d066a-lp7cp" Jul 26 16:37:03.185321 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:03.185299 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79c529d-ed3e-432a-96d5-544fbdd722ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6f5d066a-lp7cp\" (UID: \"a79c529d-ed3e-432a-96d5-544fbdd722ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6f5d066a-lp7cp" Jul 26 16:37:03.376476 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:03.376372 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6f5d066a-lp7cp" Jul 26 16:37:03.517036 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:03.517006 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6f5d066a-lp7cp"] Jul 26 16:37:03.518734 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:37:03.518710 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79c529d_ed3e_432a_96d5_544fbdd722ab.slice/crio-a78b6fc8d36064bfd1996203c5073818c8dc42b078c8397905bce603dfe8a783 WatchSource:0}: Error finding container a78b6fc8d36064bfd1996203c5073818c8dc42b078c8397905bce603dfe8a783: Status 404 returned error can't find the container with id a78b6fc8d36064bfd1996203c5073818c8dc42b078c8397905bce603dfe8a783 Jul 26 16:37:03.590262 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:03.590203 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6f5d066a-lp7cp" event={"ID":"a79c529d-ed3e-432a-96d5-544fbdd722ab","Type":"ContainerStarted","Data":"a78b6fc8d36064bfd1996203c5073818c8dc42b078c8397905bce603dfe8a783"} Jul 26 16:37:07.616114 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:07.616071 2563 generic.go:356] "Generic (PLEG): container finished" podID="a79c529d-ed3e-432a-96d5-544fbdd722ab" containerID="952f482f85942cd472540ce6f50f67a4e4db2bb433e28244319bf677151ff0d0" exitCode=0 Jul 26 16:37:07.616513 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:07.616158 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6f5d066a-lp7cp" event={"ID":"a79c529d-ed3e-432a-96d5-544fbdd722ab","Type":"ContainerDied","Data":"952f482f85942cd472540ce6f50f67a4e4db2bb433e28244319bf677151ff0d0"} Jul 26 16:37:08.751184 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:08.751157 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6f5d066a-lp7cp" Jul 26 16:37:08.832339 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:08.832299 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79c529d-ed3e-432a-96d5-544fbdd722ab-maas-api-service-ca\") pod \"a79c529d-ed3e-432a-96d5-544fbdd722ab\" (UID: \"a79c529d-ed3e-432a-96d5-544fbdd722ab\") " Jul 26 16:37:08.832638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:08.832615 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79c529d-ed3e-432a-96d5-544fbdd722ab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a79c529d-ed3e-432a-96d5-544fbdd722ab" (UID: "a79c529d-ed3e-432a-96d5-544fbdd722ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:37:08.933355 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:08.933321 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79c529d-ed3e-432a-96d5-544fbdd722ab-maas-api-service-ca\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:37:09.627023 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:09.626986 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6f5d066a-lp7cp" event={"ID":"a79c529d-ed3e-432a-96d5-544fbdd722ab","Type":"ContainerDied","Data":"a78b6fc8d36064bfd1996203c5073818c8dc42b078c8397905bce603dfe8a783"} Jul 26 16:37:09.627198 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:09.627038 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a78b6fc8d36064bfd1996203c5073818c8dc42b078c8397905bce603dfe8a783" Jul 26 16:37:09.627198 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:09.626996 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6f5d066a-lp7cp" Jul 26 16:37:18.127564 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:18.127526 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6f5d066a-lp7cp"] Jul 26 16:37:18.129493 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:18.129462 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6f5d066a-lp7cp"] Jul 26 16:37:18.264419 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:18.264386 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh"] Jul 26 16:37:18.264667 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:18.264639 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" podUID="5712ffe2-1838-4e97-ba55-2a914d43f040" containerName="istio-proxy" containerID="cri-o://e25add81e7201a7f8655af29890dc73dccea497d28c15e85d48f6b363d909f73" gracePeriod=30 Jul 26 16:37:18.890123 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:18.890091 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a79c529d-ed3e-432a-96d5-544fbdd722ab" path="/var/lib/kubelet/pods/a79c529d-ed3e-432a-96d5-544fbdd722ab/volumes" Jul 26 16:37:19.041377 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.041343 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4"] Jul 26 16:37:19.041716 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.041703 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c185a8c7-d142-4be0-9e47-e051555ebdec" containerName="istio-proxy" Jul 26 16:37:19.041761 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.041718 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c185a8c7-d142-4be0-9e47-e051555ebdec" containerName="istio-proxy" Jul 26 16:37:19.041761 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.041741 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a79c529d-ed3e-432a-96d5-544fbdd722ab" containerName="revoke-keys" Jul 26 16:37:19.041761 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.041746 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a79c529d-ed3e-432a-96d5-544fbdd722ab" containerName="revoke-keys" Jul 26 16:37:19.041856 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.041799 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c185a8c7-d142-4be0-9e47-e051555ebdec" containerName="istio-proxy" Jul 26 16:37:19.041856 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.041808 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a79c529d-ed3e-432a-96d5-544fbdd722ab" containerName="revoke-keys" Jul 26 16:37:19.044943 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.044917 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.047634 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.047613 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9f8fd10f-openshift-default-dockercfg-fjdf5\"" Jul 26 16:37:19.060300 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.060274 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4"] Jul 26 16:37:19.126489 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.126447 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-workload-certs\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.126681 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.126524 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-workload-socket\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.126681 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.126592 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-data\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.126764 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.126705 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-podinfo\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.126764 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.126744 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j4zd\" (UniqueName: \"kubernetes.io/projected/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-kube-api-access-4j4zd\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.126842 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.126821 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-credential-socket\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.126886 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.126859 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-envoy\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.126932 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.126886 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-token\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.127097 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.126982 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istiod-ca-cert\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.228517 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.228428 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-credential-socket\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.228517 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.228467 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-envoy\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.228517 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.228483 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-token\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.229067 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.228586 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istiod-ca-cert\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.229067 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.228615 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-workload-certs\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.229067 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.228650 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-workload-socket\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.229067 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.228819 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-data\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.229067 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.228867 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-podinfo\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.229067 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.228883 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-credential-socket\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.229067 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.228897 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4zd\" (UniqueName: \"kubernetes.io/projected/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-kube-api-access-4j4zd\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.229067 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.229019 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-workload-certs\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.229404 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.229128 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-workload-socket\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.229404 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.229146 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-data\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.229488 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.229460 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istiod-ca-cert\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.231044 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.231015 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-envoy\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.231241 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.231208 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-podinfo\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.237015 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.236991 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j4zd\" (UniqueName: \"kubernetes.io/projected/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-kube-api-access-4j4zd\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.237178 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.237154 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-token\") pod \"e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.354505 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.354468 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:19.501425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.501400 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4"] Jul 26 16:37:19.503290 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:37:19.503255 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6e5410_904c_43ca_b3d0_dcd7b30a82bc.slice/crio-e52cb58bdf29be4b461799ec9a083b2be31329f904dd29ea16463ace10812669 WatchSource:0}: Error finding container e52cb58bdf29be4b461799ec9a083b2be31329f904dd29ea16463ace10812669: Status 404 returned error can't find the container with id e52cb58bdf29be4b461799ec9a083b2be31329f904dd29ea16463ace10812669 Jul 26 16:37:19.505615 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.505586 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:37:19.505690 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.505647 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:37:19.505690 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.505676 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:37:19.679680 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.679642 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" event={"ID":"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc","Type":"ContainerStarted","Data":"58b0af7a16e48b161c7e204e81dcbd798b136f4fd300679bb3e2a98d085a6ec0"} Jul 26 16:37:19.679850 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.679704 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" event={"ID":"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc","Type":"ContainerStarted","Data":"e52cb58bdf29be4b461799ec9a083b2be31329f904dd29ea16463ace10812669"} Jul 26 16:37:19.702437 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:19.702367 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" podStartSLOduration=0.702347483 podStartE2EDuration="702.347483ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:37:19.701861085 +0000 UTC m=+1713.282947687" watchObservedRunningTime="2026-07-26 16:37:19.702347483 +0000 UTC m=+1713.283434054" Jul 26 16:37:20.354928 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:20.354883 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:20.360520 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:20.360490 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:20.684579 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:20.684492 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:20.685533 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:20.685511 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:22.758802 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:22.758767 2563 patch_prober.go:28] interesting pod/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 16:37:22.759324 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:22.758826 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" podUID="5712ffe2-1838-4e97-ba55-2a914d43f040" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 16:37:23.512758 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.512734 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:37:23.572276 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.572246 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs2nt\" (UniqueName: \"kubernetes.io/projected/5712ffe2-1838-4e97-ba55-2a914d43f040-kube-api-access-qs2nt\") pod \"5712ffe2-1838-4e97-ba55-2a914d43f040\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " Jul 26 16:37:23.572452 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.572334 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-token\") pod \"5712ffe2-1838-4e97-ba55-2a914d43f040\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " Jul 26 16:37:23.572452 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.572373 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-workload-certs\") pod \"5712ffe2-1838-4e97-ba55-2a914d43f040\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " Jul 26 16:37:23.572452 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.572404 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-envoy\") pod \"5712ffe2-1838-4e97-ba55-2a914d43f040\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " Jul 26 16:37:23.572452 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.572426 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-workload-socket\") pod \"5712ffe2-1838-4e97-ba55-2a914d43f040\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " Jul 26 16:37:23.572643 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.572489 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-data\") pod \"5712ffe2-1838-4e97-ba55-2a914d43f040\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " Jul 26 16:37:23.572643 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.572527 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-credential-socket\") pod \"5712ffe2-1838-4e97-ba55-2a914d43f040\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " Jul 26 16:37:23.572643 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.572560 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-podinfo\") pod \"5712ffe2-1838-4e97-ba55-2a914d43f040\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " Jul 26 16:37:23.572643 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.572583 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5712ffe2-1838-4e97-ba55-2a914d43f040-istiod-ca-cert\") pod \"5712ffe2-1838-4e97-ba55-2a914d43f040\" (UID: \"5712ffe2-1838-4e97-ba55-2a914d43f040\") " Jul 26 16:37:23.572856 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.572701 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5712ffe2-1838-4e97-ba55-2a914d43f040" (UID: "5712ffe2-1838-4e97-ba55-2a914d43f040"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:37:23.572856 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.572785 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5712ffe2-1838-4e97-ba55-2a914d43f040" (UID: "5712ffe2-1838-4e97-ba55-2a914d43f040"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:37:23.573001 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.572970 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5712ffe2-1838-4e97-ba55-2a914d43f040" (UID: "5712ffe2-1838-4e97-ba55-2a914d43f040"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:37:23.573132 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.572987 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-workload-certs\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:37:23.573132 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.573037 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-workload-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:37:23.573275 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.573256 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5712ffe2-1838-4e97-ba55-2a914d43f040-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5712ffe2-1838-4e97-ba55-2a914d43f040" (UID: "5712ffe2-1838-4e97-ba55-2a914d43f040"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:37:23.573981 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.573949 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-data" (OuterVolumeSpecName: "istio-data") pod "5712ffe2-1838-4e97-ba55-2a914d43f040" (UID: "5712ffe2-1838-4e97-ba55-2a914d43f040"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:37:23.574582 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.574557 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-token" (OuterVolumeSpecName: "istio-token") pod "5712ffe2-1838-4e97-ba55-2a914d43f040" (UID: "5712ffe2-1838-4e97-ba55-2a914d43f040"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:37:23.574676 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.574599 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5712ffe2-1838-4e97-ba55-2a914d43f040-kube-api-access-qs2nt" (OuterVolumeSpecName: "kube-api-access-qs2nt") pod "5712ffe2-1838-4e97-ba55-2a914d43f040" (UID: "5712ffe2-1838-4e97-ba55-2a914d43f040"). InnerVolumeSpecName "kube-api-access-qs2nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:37:23.574718 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.574678 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5712ffe2-1838-4e97-ba55-2a914d43f040" (UID: "5712ffe2-1838-4e97-ba55-2a914d43f040"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:37:23.574837 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.574816 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5712ffe2-1838-4e97-ba55-2a914d43f040" (UID: "5712ffe2-1838-4e97-ba55-2a914d43f040"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:37:23.674332 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.674297 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-envoy\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:37:23.674332 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.674325 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-data\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:37:23.674332 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.674335 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5712ffe2-1838-4e97-ba55-2a914d43f040-credential-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:37:23.674571 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.674345 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-podinfo\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:37:23.674571 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.674354 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5712ffe2-1838-4e97-ba55-2a914d43f040-istiod-ca-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:37:23.674571 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.674362 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs2nt\" (UniqueName: \"kubernetes.io/projected/5712ffe2-1838-4e97-ba55-2a914d43f040-kube-api-access-qs2nt\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:37:23.674571 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.674370 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5712ffe2-1838-4e97-ba55-2a914d43f040-istio-token\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:37:23.704189 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.704150 2563 generic.go:356] "Generic (PLEG): container finished" podID="5712ffe2-1838-4e97-ba55-2a914d43f040" containerID="e25add81e7201a7f8655af29890dc73dccea497d28c15e85d48f6b363d909f73" exitCode=0 Jul 26 16:37:23.704367 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.704203 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" Jul 26 16:37:23.704367 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.704200 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" event={"ID":"5712ffe2-1838-4e97-ba55-2a914d43f040","Type":"ContainerDied","Data":"e25add81e7201a7f8655af29890dc73dccea497d28c15e85d48f6b363d909f73"} Jul 26 16:37:23.704367 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.704357 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh" event={"ID":"5712ffe2-1838-4e97-ba55-2a914d43f040","Type":"ContainerDied","Data":"b5f820b36dced4bfeb9befd6fbdfbe5ad4eb5cf626b0e4bef1510fee61c86144"} Jul 26 16:37:23.704521 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.704393 2563 scope.go:117] "RemoveContainer" containerID="e25add81e7201a7f8655af29890dc73dccea497d28c15e85d48f6b363d909f73" Jul 26 16:37:23.714259 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.714240 2563 scope.go:117] "RemoveContainer" containerID="879687ead7a00af9a3d2271462aaa3ed448eda1ddb516e52ef9c42a2a4d048db" Jul 26 16:37:23.722755 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.722736 2563 scope.go:117] "RemoveContainer" containerID="e25add81e7201a7f8655af29890dc73dccea497d28c15e85d48f6b363d909f73" Jul 26 16:37:23.722991 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:37:23.722973 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25add81e7201a7f8655af29890dc73dccea497d28c15e85d48f6b363d909f73\": container with ID starting with e25add81e7201a7f8655af29890dc73dccea497d28c15e85d48f6b363d909f73 not found: ID does not exist" containerID="e25add81e7201a7f8655af29890dc73dccea497d28c15e85d48f6b363d909f73" Jul 26 16:37:23.723038 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.723001 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25add81e7201a7f8655af29890dc73dccea497d28c15e85d48f6b363d909f73"} err="failed to get container status \"e25add81e7201a7f8655af29890dc73dccea497d28c15e85d48f6b363d909f73\": rpc error: code = NotFound desc = could not find container \"e25add81e7201a7f8655af29890dc73dccea497d28c15e85d48f6b363d909f73\": container with ID starting with e25add81e7201a7f8655af29890dc73dccea497d28c15e85d48f6b363d909f73 not found: ID does not exist" Jul 26 16:37:23.723038 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.723018 2563 scope.go:117] "RemoveContainer" containerID="879687ead7a00af9a3d2271462aaa3ed448eda1ddb516e52ef9c42a2a4d048db" Jul 26 16:37:23.723262 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:37:23.723240 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"879687ead7a00af9a3d2271462aaa3ed448eda1ddb516e52ef9c42a2a4d048db\": container with ID starting with 879687ead7a00af9a3d2271462aaa3ed448eda1ddb516e52ef9c42a2a4d048db not found: ID does not exist" containerID="879687ead7a00af9a3d2271462aaa3ed448eda1ddb516e52ef9c42a2a4d048db" Jul 26 16:37:23.723312 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.723270 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"879687ead7a00af9a3d2271462aaa3ed448eda1ddb516e52ef9c42a2a4d048db"} err="failed to get container status \"879687ead7a00af9a3d2271462aaa3ed448eda1ddb516e52ef9c42a2a4d048db\": rpc error: code = NotFound desc = could not find container \"879687ead7a00af9a3d2271462aaa3ed448eda1ddb516e52ef9c42a2a4d048db\": container with ID starting with 879687ead7a00af9a3d2271462aaa3ed448eda1ddb516e52ef9c42a2a4d048db not found: ID does not exist" Jul 26 16:37:23.727994 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.727965 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh"] Jul 26 16:37:23.733930 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:23.733902 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-6f5d066a-gw-openshift-default-6d4c67fdb8-fc9mh"] Jul 26 16:37:24.890898 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:24.890866 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5712ffe2-1838-4e97-ba55-2a914d43f040" path="/var/lib/kubelet/pods/5712ffe2-1838-4e97-ba55-2a914d43f040/volumes" Jul 26 16:37:35.685611 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:35.685574 2563 patch_prober.go:28] interesting pod/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 26 16:37:35.686016 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:35.685650 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" podUID="bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 26 16:37:35.776566 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:35.776531 2563 generic.go:356] "Generic (PLEG): container finished" podID="bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" containerID="58b0af7a16e48b161c7e204e81dcbd798b136f4fd300679bb3e2a98d085a6ec0" exitCode=0 Jul 26 16:37:35.776727 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:35.776604 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" event={"ID":"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc","Type":"ContainerDied","Data":"58b0af7a16e48b161c7e204e81dcbd798b136f4fd300679bb3e2a98d085a6ec0"} Jul 26 16:37:35.777070 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:35.777052 2563 scope.go:117] "RemoveContainer" containerID="58b0af7a16e48b161c7e204e81dcbd798b136f4fd300679bb3e2a98d085a6ec0" Jul 26 16:37:35.778114 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:35.778087 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:37:35.778178 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:35.778156 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:37:35.778214 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:35.778191 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:37:36.786886 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:36.786849 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" event={"ID":"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc","Type":"ContainerStarted","Data":"6e97309891ea9fd702f7886c5da4faebdad9d8aff827951d70650be378ef6f87"} Jul 26 16:37:37.354930 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:37.354890 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:37.355209 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:37.355184 2563 patch_prober.go:28] interesting pod/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 26 16:37:37.355303 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:37.355260 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" podUID="bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 26 16:37:38.355640 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:38.355601 2563 patch_prober.go:28] interesting pod/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 26 16:37:38.356049 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:38.355657 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" podUID="bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 26 16:37:39.355400 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:39.355361 2563 patch_prober.go:28] interesting pod/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 26 16:37:39.355574 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:39.355421 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" podUID="bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 26 16:37:40.355968 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:40.355928 2563 patch_prober.go:28] interesting pod/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 26 16:37:40.356435 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:40.356004 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" podUID="bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 26 16:37:41.358373 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:41.358336 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:41.358780 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:41.358691 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:41.359502 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:41.359484 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:37:46.892385 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:46.892353 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc"] Jul 26 16:37:46.892731 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:46.892666 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5712ffe2-1838-4e97-ba55-2a914d43f040" containerName="istio-proxy" Jul 26 16:37:46.892731 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:46.892677 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5712ffe2-1838-4e97-ba55-2a914d43f040" containerName="istio-proxy" Jul 26 16:37:46.892731 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:46.892696 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5712ffe2-1838-4e97-ba55-2a914d43f040" containerName="istio-proxy" Jul 26 16:37:46.892731 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:46.892701 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5712ffe2-1838-4e97-ba55-2a914d43f040" containerName="istio-proxy" Jul 26 16:37:46.892878 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:46.892758 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5712ffe2-1838-4e97-ba55-2a914d43f040" containerName="istio-proxy" Jul 26 16:37:46.892878 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:46.892766 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5712ffe2-1838-4e97-ba55-2a914d43f040" containerName="istio-proxy" Jul 26 16:37:46.900488 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:46.900460 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc"] Jul 26 16:37:46.900633 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:46.900574 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc" Jul 26 16:37:46.903495 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:46.903470 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2xlcn\"" Jul 26 16:37:47.089072 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:47.089028 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ab3547b-051d-40f7-ad8d-78c859c97122-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc\" (UID: \"4ab3547b-051d-40f7-ad8d-78c859c97122\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc" Jul 26 16:37:47.190397 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:47.190302 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ab3547b-051d-40f7-ad8d-78c859c97122-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc\" (UID: \"4ab3547b-051d-40f7-ad8d-78c859c97122\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc" Jul 26 16:37:47.190945 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:47.190923 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ab3547b-051d-40f7-ad8d-78c859c97122-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc\" (UID: \"4ab3547b-051d-40f7-ad8d-78c859c97122\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc" Jul 26 16:37:47.210795 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:47.210766 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc" Jul 26 16:37:47.355266 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:47.355208 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc"] Jul 26 16:37:47.356805 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:37:47.356775 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab3547b_051d_40f7_ad8d_78c859c97122.slice/crio-1ca6b573dd64b7d17163db71e9db0af1244f93679c8f2331e4cfe5992e65c240 WatchSource:0}: Error finding container 1ca6b573dd64b7d17163db71e9db0af1244f93679c8f2331e4cfe5992e65c240: Status 404 returned error can't find the container with id 1ca6b573dd64b7d17163db71e9db0af1244f93679c8f2331e4cfe5992e65c240 Jul 26 16:37:47.847820 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:47.847725 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc" event={"ID":"4ab3547b-051d-40f7-ad8d-78c859c97122","Type":"ContainerStarted","Data":"35b9fe21e958145aadf59aeba76af7449969cea4bbfea255fdb7233651556b48"} Jul 26 16:37:47.847820 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:47.847782 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc" event={"ID":"4ab3547b-051d-40f7-ad8d-78c859c97122","Type":"ContainerStarted","Data":"1ca6b573dd64b7d17163db71e9db0af1244f93679c8f2331e4cfe5992e65c240"} Jul 26 16:37:47.864320 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:47.864257 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc" podStartSLOduration=1.864241445 podStartE2EDuration="1.864241445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:37:47.86212177 +0000 UTC m=+1741.443208338" watchObservedRunningTime="2026-07-26 16:37:47.864241445 +0000 UTC m=+1741.445328012" Jul 26 16:37:48.854273 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:48.854235 2563 generic.go:356] "Generic (PLEG): container finished" podID="4ab3547b-051d-40f7-ad8d-78c859c97122" containerID="35b9fe21e958145aadf59aeba76af7449969cea4bbfea255fdb7233651556b48" exitCode=0 Jul 26 16:37:48.854660 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:48.854321 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc" event={"ID":"4ab3547b-051d-40f7-ad8d-78c859c97122","Type":"ContainerDied","Data":"35b9fe21e958145aadf59aeba76af7449969cea4bbfea255fdb7233651556b48"} Jul 26 16:37:49.994262 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:49.994211 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc" Jul 26 16:37:50.114772 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:50.114680 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ab3547b-051d-40f7-ad8d-78c859c97122-maas-api-service-ca\") pod \"4ab3547b-051d-40f7-ad8d-78c859c97122\" (UID: \"4ab3547b-051d-40f7-ad8d-78c859c97122\") " Jul 26 16:37:50.115071 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:50.115042 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab3547b-051d-40f7-ad8d-78c859c97122-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4ab3547b-051d-40f7-ad8d-78c859c97122" (UID: "4ab3547b-051d-40f7-ad8d-78c859c97122"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:37:50.216284 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:50.216216 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ab3547b-051d-40f7-ad8d-78c859c97122-maas-api-service-ca\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:37:50.865642 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:50.865611 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc" Jul 26 16:37:50.865847 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:50.865644 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc" event={"ID":"4ab3547b-051d-40f7-ad8d-78c859c97122","Type":"ContainerDied","Data":"1ca6b573dd64b7d17163db71e9db0af1244f93679c8f2331e4cfe5992e65c240"} Jul 26 16:37:50.865847 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:37:50.865686 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ca6b573dd64b7d17163db71e9db0af1244f93679c8f2331e4cfe5992e65c240" Jul 26 16:38:01.968927 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:01.968867 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc"] Jul 26 16:38:01.971201 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:01.971171 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f8fd10f-42jjc"] Jul 26 16:38:02.890772 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:02.890740 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ab3547b-051d-40f7-ad8d-78c859c97122" path="/var/lib/kubelet/pods/4ab3547b-051d-40f7-ad8d-78c859c97122/volumes" Jul 26 16:38:03.094096 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:03.094063 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4"] Jul 26 16:38:03.094561 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:03.094293 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" podUID="bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" containerName="istio-proxy" containerID="cri-o://6e97309891ea9fd702f7886c5da4faebdad9d8aff827951d70650be378ef6f87" gracePeriod=30 Jul 26 16:38:03.860066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:03.860035 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr"] Jul 26 16:38:03.860586 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:03.860561 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ab3547b-051d-40f7-ad8d-78c859c97122" containerName="revoke-keys" Jul 26 16:38:03.860586 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:03.860583 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ab3547b-051d-40f7-ad8d-78c859c97122" containerName="revoke-keys" Jul 26 16:38:03.860808 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:03.860657 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ab3547b-051d-40f7-ad8d-78c859c97122" containerName="revoke-keys" Jul 26 16:38:03.863878 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:03.863856 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:03.867573 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:03.867548 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2774653d-openshift-default-dockercfg-nwdkc\"" Jul 26 16:38:03.881075 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:03.881053 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr"] Jul 26 16:38:03.938724 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:03.938552 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-workload-socket\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:03.938724 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:03.938595 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-envoy\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:03.938724 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:03.938638 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-data\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:03.938724 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:03.938659 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istiod-ca-cert\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:03.939091 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:03.938844 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-token\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:03.939091 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:03.938926 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-podinfo\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:03.939091 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:03.939014 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-workload-certs\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:03.939091 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:03.939056 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-credential-socket\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:03.939091 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:03.939088 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbqgp\" (UniqueName: \"kubernetes.io/projected/b967e2cc-9c99-4654-88ce-ee42fb19fe99-kube-api-access-vbqgp\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.040605 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.040556 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-data\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.040605 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.040600 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istiod-ca-cert\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.040878 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.040654 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-token\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.040878 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.040693 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-podinfo\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.040878 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.040727 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-workload-certs\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.040878 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.040758 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-credential-socket\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.040878 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.040786 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbqgp\" (UniqueName: \"kubernetes.io/projected/b967e2cc-9c99-4654-88ce-ee42fb19fe99-kube-api-access-vbqgp\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.040878 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.040867 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-workload-socket\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.041177 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.040895 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-envoy\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.041177 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.041044 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-data\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.041177 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.041092 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-workload-certs\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.041360 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.041190 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-credential-socket\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.041360 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.041345 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-workload-socket\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.041533 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.041512 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istiod-ca-cert\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.043121 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.043100 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-envoy\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.043272 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.043256 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-podinfo\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.052807 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.052781 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-token\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.057236 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.057197 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbqgp\" (UniqueName: \"kubernetes.io/projected/b967e2cc-9c99-4654-88ce-ee42fb19fe99-kube-api-access-vbqgp\") pod \"e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.174143 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.174050 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:04.323787 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.323760 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr"] Jul 26 16:38:04.326012 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:38:04.325986 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb967e2cc_9c99_4654_88ce_ee42fb19fe99.slice/crio-e1db17b213fa226ebc5a9753b3d9a01defde0f9c6778b108e790c8b442e0be75 WatchSource:0}: Error finding container e1db17b213fa226ebc5a9753b3d9a01defde0f9c6778b108e790c8b442e0be75: Status 404 returned error can't find the container with id e1db17b213fa226ebc5a9753b3d9a01defde0f9c6778b108e790c8b442e0be75 Jul 26 16:38:04.328069 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.328041 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:38:04.328138 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.328100 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:38:04.328138 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.328127 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:38:04.941156 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.941113 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" event={"ID":"b967e2cc-9c99-4654-88ce-ee42fb19fe99","Type":"ContainerStarted","Data":"4c750cbe21e62c72e1f7a858fe0ec83007fccd0d6836018a531b97766edba1ab"} Jul 26 16:38:04.941362 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:04.941179 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" event={"ID":"b967e2cc-9c99-4654-88ce-ee42fb19fe99","Type":"ContainerStarted","Data":"e1db17b213fa226ebc5a9753b3d9a01defde0f9c6778b108e790c8b442e0be75"} Jul 26 16:38:05.174288 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:05.174245 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:05.179256 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:05.179206 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:05.200673 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:05.200565 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" podStartSLOduration=2.200548235 podStartE2EDuration="2.200548235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:38:04.9631689 +0000 UTC m=+1758.544255477" watchObservedRunningTime="2026-07-26 16:38:05.200548235 +0000 UTC m=+1758.781634864" Jul 26 16:38:05.946197 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:05.946166 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:05.947244 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:05.947197 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:06.512756 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:06.512721 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9"] Jul 26 16:38:06.516540 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:06.516514 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" Jul 26 16:38:06.519473 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:06.519450 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-2774653d-serving-cert\"" Jul 26 16:38:06.530536 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:06.530510 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9"] Jul 26 16:38:06.564038 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:06.563998 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9ce8931-d65c-4247-9475-ade8884e9db4-maas-api-tls\") pod \"maas-api-e2e-derive-2774653d-576789ff97-scnt9\" (UID: \"b9ce8931-d65c-4247-9475-ade8884e9db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" Jul 26 16:38:06.564427 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:06.564393 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ce8931-d65c-4247-9475-ade8884e9db4-openshift-service-ca\") pod \"maas-api-e2e-derive-2774653d-576789ff97-scnt9\" (UID: \"b9ce8931-d65c-4247-9475-ade8884e9db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" Jul 26 16:38:06.564544 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:06.564502 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j777w\" (UniqueName: \"kubernetes.io/projected/b9ce8931-d65c-4247-9475-ade8884e9db4-kube-api-access-j777w\") pod \"maas-api-e2e-derive-2774653d-576789ff97-scnt9\" (UID: \"b9ce8931-d65c-4247-9475-ade8884e9db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" Jul 26 16:38:06.665741 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:06.665694 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ce8931-d65c-4247-9475-ade8884e9db4-openshift-service-ca\") pod \"maas-api-e2e-derive-2774653d-576789ff97-scnt9\" (UID: \"b9ce8931-d65c-4247-9475-ade8884e9db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" Jul 26 16:38:06.665921 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:06.665753 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j777w\" (UniqueName: \"kubernetes.io/projected/b9ce8931-d65c-4247-9475-ade8884e9db4-kube-api-access-j777w\") pod \"maas-api-e2e-derive-2774653d-576789ff97-scnt9\" (UID: \"b9ce8931-d65c-4247-9475-ade8884e9db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" Jul 26 16:38:06.665921 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:06.665804 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9ce8931-d65c-4247-9475-ade8884e9db4-maas-api-tls\") pod \"maas-api-e2e-derive-2774653d-576789ff97-scnt9\" (UID: \"b9ce8931-d65c-4247-9475-ade8884e9db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" Jul 26 16:38:06.665921 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:38:06.665899 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-serving-cert: secret "maas-api-e2e-derive-2774653d-serving-cert" not found Jul 26 16:38:06.666112 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:38:06.665958 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9ce8931-d65c-4247-9475-ade8884e9db4-maas-api-tls podName:b9ce8931-d65c-4247-9475-ade8884e9db4 nodeName:}" failed. No retries permitted until 2026-07-26 16:38:07.165942549 +0000 UTC m=+1760.747029097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9ce8931-d65c-4247-9475-ade8884e9db4-maas-api-tls") pod "maas-api-e2e-derive-2774653d-576789ff97-scnt9" (UID: "b9ce8931-d65c-4247-9475-ade8884e9db4") : secret "maas-api-e2e-derive-2774653d-serving-cert" not found Jul 26 16:38:06.666403 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:06.666380 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ce8931-d65c-4247-9475-ade8884e9db4-openshift-service-ca\") pod \"maas-api-e2e-derive-2774653d-576789ff97-scnt9\" (UID: \"b9ce8931-d65c-4247-9475-ade8884e9db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" Jul 26 16:38:06.675451 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:06.675429 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j777w\" (UniqueName: \"kubernetes.io/projected/b9ce8931-d65c-4247-9475-ade8884e9db4-kube-api-access-j777w\") pod \"maas-api-e2e-derive-2774653d-576789ff97-scnt9\" (UID: \"b9ce8931-d65c-4247-9475-ade8884e9db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" Jul 26 16:38:07.170798 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:07.170760 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9ce8931-d65c-4247-9475-ade8884e9db4-maas-api-tls\") pod \"maas-api-e2e-derive-2774653d-576789ff97-scnt9\" (UID: \"b9ce8931-d65c-4247-9475-ade8884e9db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" Jul 26 16:38:07.173415 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:07.173387 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9ce8931-d65c-4247-9475-ade8884e9db4-maas-api-tls\") pod \"maas-api-e2e-derive-2774653d-576789ff97-scnt9\" (UID: \"b9ce8931-d65c-4247-9475-ade8884e9db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" Jul 26 16:38:07.428274 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:07.428158 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" Jul 26 16:38:07.770876 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:07.770845 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9"] Jul 26 16:38:07.772762 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:38:07.772733 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ce8931_d65c_4247_9475_ade8884e9db4.slice/crio-4cccef8eef0ad4a0b11da519ea25019cb8b69c3e2faeb3aee4d61777c0460494 WatchSource:0}: Error finding container 4cccef8eef0ad4a0b11da519ea25019cb8b69c3e2faeb3aee4d61777c0460494: Status 404 returned error can't find the container with id 4cccef8eef0ad4a0b11da519ea25019cb8b69c3e2faeb3aee4d61777c0460494 Jul 26 16:38:07.957587 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:07.957552 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" event={"ID":"b9ce8931-d65c-4247-9475-ade8884e9db4","Type":"ContainerStarted","Data":"4cccef8eef0ad4a0b11da519ea25019cb8b69c3e2faeb3aee4d61777c0460494"} Jul 26 16:38:08.358102 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.358076 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:38:08.482586 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.482495 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-token\") pod \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " Jul 26 16:38:08.482586 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.482538 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-credential-socket\") pod \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " Jul 26 16:38:08.482586 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.482584 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-envoy\") pod \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " Jul 26 16:38:08.482829 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.482740 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istiod-ca-cert\") pod \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " Jul 26 16:38:08.482829 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.482777 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-data\") pod \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " Jul 26 16:38:08.482829 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.482820 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-podinfo\") pod \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " Jul 26 16:38:08.483068 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.482861 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" (UID: "bf6e5410-904c-43ca-b3d0-dcd7b30a82bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:38:08.483068 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.482906 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-workload-socket\") pod \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " Jul 26 16:38:08.483068 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.482952 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-workload-certs\") pod \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " Jul 26 16:38:08.483068 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.482985 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4j4zd\" (UniqueName: \"kubernetes.io/projected/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-kube-api-access-4j4zd\") pod \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\" (UID: \"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc\") " Jul 26 16:38:08.483315 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.483157 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" (UID: "bf6e5410-904c-43ca-b3d0-dcd7b30a82bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:38:08.483315 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.483200 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" (UID: "bf6e5410-904c-43ca-b3d0-dcd7b30a82bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:38:08.483479 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.483370 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-credential-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:08.483479 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.483461 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" (UID: "bf6e5410-904c-43ca-b3d0-dcd7b30a82bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:38:08.483479 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.483465 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istiod-ca-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:08.483661 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.483516 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-workload-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:08.483789 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.483761 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" (UID: "bf6e5410-904c-43ca-b3d0-dcd7b30a82bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:38:08.484925 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.484903 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" (UID: "bf6e5410-904c-43ca-b3d0-dcd7b30a82bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:38:08.485032 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.484941 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" (UID: "bf6e5410-904c-43ca-b3d0-dcd7b30a82bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:38:08.485150 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.485135 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" (UID: "bf6e5410-904c-43ca-b3d0-dcd7b30a82bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:38:08.485341 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.485325 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-kube-api-access-4j4zd" (OuterVolumeSpecName: "kube-api-access-4j4zd") pod "bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" (UID: "bf6e5410-904c-43ca-b3d0-dcd7b30a82bc"). InnerVolumeSpecName "kube-api-access-4j4zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:38:08.584561 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.584530 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-envoy\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:08.584561 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.584557 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-data\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:08.584561 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.584566 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-podinfo\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:08.584799 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.584574 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-workload-certs\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:08.584799 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.584584 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j4zd\" (UniqueName: \"kubernetes.io/projected/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-kube-api-access-4j4zd\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:08.584799 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.584594 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc-istio-token\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:08.971650 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.971612 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" event={"ID":"b9ce8931-d65c-4247-9475-ade8884e9db4","Type":"ContainerStarted","Data":"f939304d459a60d9b3001391835664f6c7bad50c1656391de4a7b64cf74cb92a"} Jul 26 16:38:08.972147 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.971710 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" Jul 26 16:38:08.973478 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.973448 2563 generic.go:356] "Generic (PLEG): container finished" podID="bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" containerID="6e97309891ea9fd702f7886c5da4faebdad9d8aff827951d70650be378ef6f87" exitCode=0 Jul 26 16:38:08.973617 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.973511 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" Jul 26 16:38:08.973617 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.973517 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" event={"ID":"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc","Type":"ContainerDied","Data":"6e97309891ea9fd702f7886c5da4faebdad9d8aff827951d70650be378ef6f87"} Jul 26 16:38:08.973617 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.973561 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4" event={"ID":"bf6e5410-904c-43ca-b3d0-dcd7b30a82bc","Type":"ContainerDied","Data":"e52cb58bdf29be4b461799ec9a083b2be31329f904dd29ea16463ace10812669"} Jul 26 16:38:08.973617 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.973583 2563 scope.go:117] "RemoveContainer" containerID="6e97309891ea9fd702f7886c5da4faebdad9d8aff827951d70650be378ef6f87" Jul 26 16:38:08.982636 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.982616 2563 scope.go:117] "RemoveContainer" containerID="58b0af7a16e48b161c7e204e81dcbd798b136f4fd300679bb3e2a98d085a6ec0" Jul 26 16:38:08.990421 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.990404 2563 scope.go:117] "RemoveContainer" containerID="6e97309891ea9fd702f7886c5da4faebdad9d8aff827951d70650be378ef6f87" Jul 26 16:38:08.990671 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:38:08.990651 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e97309891ea9fd702f7886c5da4faebdad9d8aff827951d70650be378ef6f87\": container with ID starting with 6e97309891ea9fd702f7886c5da4faebdad9d8aff827951d70650be378ef6f87 not found: ID does not exist" containerID="6e97309891ea9fd702f7886c5da4faebdad9d8aff827951d70650be378ef6f87" Jul 26 16:38:08.990750 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.990677 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e97309891ea9fd702f7886c5da4faebdad9d8aff827951d70650be378ef6f87"} err="failed to get container status \"6e97309891ea9fd702f7886c5da4faebdad9d8aff827951d70650be378ef6f87\": rpc error: code = NotFound desc = could not find container \"6e97309891ea9fd702f7886c5da4faebdad9d8aff827951d70650be378ef6f87\": container with ID starting with 6e97309891ea9fd702f7886c5da4faebdad9d8aff827951d70650be378ef6f87 not found: ID does not exist" Jul 26 16:38:08.990750 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.990698 2563 scope.go:117] "RemoveContainer" containerID="58b0af7a16e48b161c7e204e81dcbd798b136f4fd300679bb3e2a98d085a6ec0" Jul 26 16:38:08.990955 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:38:08.990936 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b0af7a16e48b161c7e204e81dcbd798b136f4fd300679bb3e2a98d085a6ec0\": container with ID starting with 58b0af7a16e48b161c7e204e81dcbd798b136f4fd300679bb3e2a98d085a6ec0 not found: ID does not exist" containerID="58b0af7a16e48b161c7e204e81dcbd798b136f4fd300679bb3e2a98d085a6ec0" Jul 26 16:38:08.991003 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.990961 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b0af7a16e48b161c7e204e81dcbd798b136f4fd300679bb3e2a98d085a6ec0"} err="failed to get container status \"58b0af7a16e48b161c7e204e81dcbd798b136f4fd300679bb3e2a98d085a6ec0\": rpc error: code = NotFound desc = could not find container \"58b0af7a16e48b161c7e204e81dcbd798b136f4fd300679bb3e2a98d085a6ec0\": container with ID starting with 58b0af7a16e48b161c7e204e81dcbd798b136f4fd300679bb3e2a98d085a6ec0 not found: ID does not exist" Jul 26 16:38:08.994580 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:08.994538 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" podStartSLOduration=2.65227977 podStartE2EDuration="2.994525413s" totalImagesPullingTime="342.245643ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:38:08.993025604 +0000 UTC m=+1762.574112175" watchObservedRunningTime="2026-07-26 16:38:08.994525413 +0000 UTC m=+1762.575611982" Jul 26 16:38:09.009077 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:09.009041 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4"] Jul 26 16:38:09.011885 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:09.011865 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9f8fd10f-openshift-default-65bcd75c6d-82qp4"] Jul 26 16:38:10.891018 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:10.890979 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" path="/var/lib/kubelet/pods/bf6e5410-904c-43ca-b3d0-dcd7b30a82bc/volumes" Jul 26 16:38:14.990574 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:14.990541 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" Jul 26 16:38:20.042235 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:20.042190 2563 generic.go:356] "Generic (PLEG): container finished" podID="b967e2cc-9c99-4654-88ce-ee42fb19fe99" containerID="4c750cbe21e62c72e1f7a858fe0ec83007fccd0d6836018a531b97766edba1ab" exitCode=0 Jul 26 16:38:20.042676 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:20.042270 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" event={"ID":"b967e2cc-9c99-4654-88ce-ee42fb19fe99","Type":"ContainerDied","Data":"4c750cbe21e62c72e1f7a858fe0ec83007fccd0d6836018a531b97766edba1ab"} Jul 26 16:38:20.042722 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:20.042665 2563 scope.go:117] "RemoveContainer" containerID="4c750cbe21e62c72e1f7a858fe0ec83007fccd0d6836018a531b97766edba1ab" Jul 26 16:38:20.043685 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:20.043656 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:38:20.043735 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:20.043723 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:38:20.043774 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:20.043757 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:38:20.946171 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:20.946127 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:21.048854 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:21.048817 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" event={"ID":"b967e2cc-9c99-4654-88ce-ee42fb19fe99","Type":"ContainerStarted","Data":"6fa0ed3055a421af890a5f6e4eb727c16e71ae4357198ab659c43ac91a020147"} Jul 26 16:38:21.174987 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:21.174953 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:21.176147 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:21.176115 2563 patch_prober.go:28] interesting pod/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 26 16:38:21.176415 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:21.176387 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" podUID="b967e2cc-9c99-4654-88ce-ee42fb19fe99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 26 16:38:22.174570 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:22.174531 2563 patch_prober.go:28] interesting pod/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 26 16:38:22.175659 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:22.175267 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" podUID="b967e2cc-9c99-4654-88ce-ee42fb19fe99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 26 16:38:23.175000 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:23.174951 2563 patch_prober.go:28] interesting pod/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 26 16:38:23.175526 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:23.175024 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" podUID="b967e2cc-9c99-4654-88ce-ee42fb19fe99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 26 16:38:24.174930 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:24.174886 2563 patch_prober.go:28] interesting pod/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 26 16:38:24.175699 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:24.175267 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" podUID="b967e2cc-9c99-4654-88ce-ee42fb19fe99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 26 16:38:25.178307 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:25.178275 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:25.178694 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:25.178638 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:25.179316 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:25.179300 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:42.098185 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:42.098103 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9"] Jul 26 16:38:42.099042 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:42.098351 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" podUID="b9ce8931-d65c-4247-9475-ade8884e9db4" containerName="maas-api" containerID="cri-o://f939304d459a60d9b3001391835664f6c7bad50c1656391de4a7b64cf74cb92a" gracePeriod=30 Jul 26 16:38:42.353018 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:42.352955 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" Jul 26 16:38:42.392552 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:42.392512 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j777w\" (UniqueName: \"kubernetes.io/projected/b9ce8931-d65c-4247-9475-ade8884e9db4-kube-api-access-j777w\") pod \"b9ce8931-d65c-4247-9475-ade8884e9db4\" (UID: \"b9ce8931-d65c-4247-9475-ade8884e9db4\") " Jul 26 16:38:42.392783 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:42.392592 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9ce8931-d65c-4247-9475-ade8884e9db4-maas-api-tls\") pod \"b9ce8931-d65c-4247-9475-ade8884e9db4\" (UID: \"b9ce8931-d65c-4247-9475-ade8884e9db4\") " Jul 26 16:38:42.392783 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:42.392627 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ce8931-d65c-4247-9475-ade8884e9db4-openshift-service-ca\") pod \"b9ce8931-d65c-4247-9475-ade8884e9db4\" (UID: \"b9ce8931-d65c-4247-9475-ade8884e9db4\") " Jul 26 16:38:42.393042 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:42.393018 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ce8931-d65c-4247-9475-ade8884e9db4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b9ce8931-d65c-4247-9475-ade8884e9db4" (UID: "b9ce8931-d65c-4247-9475-ade8884e9db4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:38:42.394763 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:42.394736 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9ce8931-d65c-4247-9475-ade8884e9db4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b9ce8931-d65c-4247-9475-ade8884e9db4" (UID: "b9ce8931-d65c-4247-9475-ade8884e9db4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:38:42.394873 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:42.394739 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ce8931-d65c-4247-9475-ade8884e9db4-kube-api-access-j777w" (OuterVolumeSpecName: "kube-api-access-j777w") pod "b9ce8931-d65c-4247-9475-ade8884e9db4" (UID: "b9ce8931-d65c-4247-9475-ade8884e9db4"). InnerVolumeSpecName "kube-api-access-j777w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:38:42.494129 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:42.494095 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j777w\" (UniqueName: \"kubernetes.io/projected/b9ce8931-d65c-4247-9475-ade8884e9db4-kube-api-access-j777w\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:42.494129 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:42.494127 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9ce8931-d65c-4247-9475-ade8884e9db4-maas-api-tls\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:42.494348 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:42.494138 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ce8931-d65c-4247-9475-ade8884e9db4-openshift-service-ca\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:43.184597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:43.184502 2563 generic.go:356] "Generic (PLEG): container finished" podID="b9ce8931-d65c-4247-9475-ade8884e9db4" containerID="f939304d459a60d9b3001391835664f6c7bad50c1656391de4a7b64cf74cb92a" exitCode=0 Jul 26 16:38:43.184597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:43.184570 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" Jul 26 16:38:43.185094 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:43.184597 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" event={"ID":"b9ce8931-d65c-4247-9475-ade8884e9db4","Type":"ContainerDied","Data":"f939304d459a60d9b3001391835664f6c7bad50c1656391de4a7b64cf74cb92a"} Jul 26 16:38:43.185094 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:43.184687 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9" event={"ID":"b9ce8931-d65c-4247-9475-ade8884e9db4","Type":"ContainerDied","Data":"4cccef8eef0ad4a0b11da519ea25019cb8b69c3e2faeb3aee4d61777c0460494"} Jul 26 16:38:43.185094 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:43.184711 2563 scope.go:117] "RemoveContainer" containerID="f939304d459a60d9b3001391835664f6c7bad50c1656391de4a7b64cf74cb92a" Jul 26 16:38:43.196399 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:43.196374 2563 scope.go:117] "RemoveContainer" containerID="f939304d459a60d9b3001391835664f6c7bad50c1656391de4a7b64cf74cb92a" Jul 26 16:38:43.196691 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:38:43.196670 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f939304d459a60d9b3001391835664f6c7bad50c1656391de4a7b64cf74cb92a\": container with ID starting with f939304d459a60d9b3001391835664f6c7bad50c1656391de4a7b64cf74cb92a not found: ID does not exist" containerID="f939304d459a60d9b3001391835664f6c7bad50c1656391de4a7b64cf74cb92a" Jul 26 16:38:43.196785 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:43.196703 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f939304d459a60d9b3001391835664f6c7bad50c1656391de4a7b64cf74cb92a"} err="failed to get container status \"f939304d459a60d9b3001391835664f6c7bad50c1656391de4a7b64cf74cb92a\": rpc error: code = NotFound desc = could not find container \"f939304d459a60d9b3001391835664f6c7bad50c1656391de4a7b64cf74cb92a\": container with ID starting with f939304d459a60d9b3001391835664f6c7bad50c1656391de4a7b64cf74cb92a not found: ID does not exist" Jul 26 16:38:43.203166 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:43.203129 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9"] Jul 26 16:38:43.206303 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:43.206278 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2774653d-576789ff97-scnt9"] Jul 26 16:38:44.891290 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:44.891193 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9ce8931-d65c-4247-9475-ade8884e9db4" path="/var/lib/kubelet/pods/b9ce8931-d65c-4247-9475-ade8884e9db4/volumes" Jul 26 16:38:46.398272 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:46.398240 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr"] Jul 26 16:38:46.398787 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:46.398521 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" podUID="b967e2cc-9c99-4654-88ce-ee42fb19fe99" containerName="istio-proxy" containerID="cri-o://6fa0ed3055a421af890a5f6e4eb727c16e71ae4357198ab659c43ac91a020147" gracePeriod=30 Jul 26 16:38:51.675152 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.675073 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:51.770033 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.770000 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-envoy\") pod \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " Jul 26 16:38:51.770033 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.770038 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-podinfo\") pod \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " Jul 26 16:38:51.770316 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.770067 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-workload-socket\") pod \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " Jul 26 16:38:51.770316 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.770093 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-credential-socket\") pod \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " Jul 26 16:38:51.770316 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.770114 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-token\") pod \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " Jul 26 16:38:51.770316 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.770161 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-workload-certs\") pod \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " Jul 26 16:38:51.770316 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.770193 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istiod-ca-cert\") pod \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " Jul 26 16:38:51.770316 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.770215 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-data\") pod \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " Jul 26 16:38:51.770316 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.770275 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbqgp\" (UniqueName: \"kubernetes.io/projected/b967e2cc-9c99-4654-88ce-ee42fb19fe99-kube-api-access-vbqgp\") pod \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\" (UID: \"b967e2cc-9c99-4654-88ce-ee42fb19fe99\") " Jul 26 16:38:51.770665 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.770396 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b967e2cc-9c99-4654-88ce-ee42fb19fe99" (UID: "b967e2cc-9c99-4654-88ce-ee42fb19fe99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:38:51.770665 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.770481 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b967e2cc-9c99-4654-88ce-ee42fb19fe99" (UID: "b967e2cc-9c99-4654-88ce-ee42fb19fe99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:38:51.770665 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.770497 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b967e2cc-9c99-4654-88ce-ee42fb19fe99" (UID: "b967e2cc-9c99-4654-88ce-ee42fb19fe99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:38:51.770665 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.770566 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b967e2cc-9c99-4654-88ce-ee42fb19fe99" (UID: "b967e2cc-9c99-4654-88ce-ee42fb19fe99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:38:51.770665 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.770646 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-workload-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:51.770665 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.770664 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-credential-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:51.770951 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.770678 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-workload-certs\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:51.770951 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.770692 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istiod-ca-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:51.771456 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.771417 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-data" (OuterVolumeSpecName: "istio-data") pod "b967e2cc-9c99-4654-88ce-ee42fb19fe99" (UID: "b967e2cc-9c99-4654-88ce-ee42fb19fe99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:38:51.773425 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.773382 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b967e2cc-9c99-4654-88ce-ee42fb19fe99-kube-api-access-vbqgp" (OuterVolumeSpecName: "kube-api-access-vbqgp") pod "b967e2cc-9c99-4654-88ce-ee42fb19fe99" (UID: "b967e2cc-9c99-4654-88ce-ee42fb19fe99"). InnerVolumeSpecName "kube-api-access-vbqgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:38:51.773717 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.773692 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b967e2cc-9c99-4654-88ce-ee42fb19fe99" (UID: "b967e2cc-9c99-4654-88ce-ee42fb19fe99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:38:51.773830 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.773751 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-token" (OuterVolumeSpecName: "istio-token") pod "b967e2cc-9c99-4654-88ce-ee42fb19fe99" (UID: "b967e2cc-9c99-4654-88ce-ee42fb19fe99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:38:51.773877 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.773830 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b967e2cc-9c99-4654-88ce-ee42fb19fe99" (UID: "b967e2cc-9c99-4654-88ce-ee42fb19fe99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:38:51.871330 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.871287 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbqgp\" (UniqueName: \"kubernetes.io/projected/b967e2cc-9c99-4654-88ce-ee42fb19fe99-kube-api-access-vbqgp\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:51.871330 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.871326 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-envoy\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:51.871330 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.871338 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-podinfo\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:51.871620 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.871348 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-token\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:51.871620 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:51.871359 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b967e2cc-9c99-4654-88ce-ee42fb19fe99-istio-data\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:38:52.236304 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:52.236269 2563 generic.go:356] "Generic (PLEG): container finished" podID="b967e2cc-9c99-4654-88ce-ee42fb19fe99" containerID="6fa0ed3055a421af890a5f6e4eb727c16e71ae4357198ab659c43ac91a020147" exitCode=0 Jul 26 16:38:52.236502 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:52.236314 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" Jul 26 16:38:52.236502 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:52.236361 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" event={"ID":"b967e2cc-9c99-4654-88ce-ee42fb19fe99","Type":"ContainerDied","Data":"6fa0ed3055a421af890a5f6e4eb727c16e71ae4357198ab659c43ac91a020147"} Jul 26 16:38:52.236502 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:52.236411 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr" event={"ID":"b967e2cc-9c99-4654-88ce-ee42fb19fe99","Type":"ContainerDied","Data":"e1db17b213fa226ebc5a9753b3d9a01defde0f9c6778b108e790c8b442e0be75"} Jul 26 16:38:52.236502 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:52.236434 2563 scope.go:117] "RemoveContainer" containerID="6fa0ed3055a421af890a5f6e4eb727c16e71ae4357198ab659c43ac91a020147" Jul 26 16:38:52.245476 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:52.245460 2563 scope.go:117] "RemoveContainer" containerID="4c750cbe21e62c72e1f7a858fe0ec83007fccd0d6836018a531b97766edba1ab" Jul 26 16:38:52.253610 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:52.253590 2563 scope.go:117] "RemoveContainer" containerID="6fa0ed3055a421af890a5f6e4eb727c16e71ae4357198ab659c43ac91a020147" Jul 26 16:38:52.253890 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:38:52.253872 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa0ed3055a421af890a5f6e4eb727c16e71ae4357198ab659c43ac91a020147\": container with ID starting with 6fa0ed3055a421af890a5f6e4eb727c16e71ae4357198ab659c43ac91a020147 not found: ID does not exist" containerID="6fa0ed3055a421af890a5f6e4eb727c16e71ae4357198ab659c43ac91a020147" Jul 26 16:38:52.253937 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:52.253899 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa0ed3055a421af890a5f6e4eb727c16e71ae4357198ab659c43ac91a020147"} err="failed to get container status \"6fa0ed3055a421af890a5f6e4eb727c16e71ae4357198ab659c43ac91a020147\": rpc error: code = NotFound desc = could not find container \"6fa0ed3055a421af890a5f6e4eb727c16e71ae4357198ab659c43ac91a020147\": container with ID starting with 6fa0ed3055a421af890a5f6e4eb727c16e71ae4357198ab659c43ac91a020147 not found: ID does not exist" Jul 26 16:38:52.253937 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:52.253915 2563 scope.go:117] "RemoveContainer" containerID="4c750cbe21e62c72e1f7a858fe0ec83007fccd0d6836018a531b97766edba1ab" Jul 26 16:38:52.254125 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:38:52.254107 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c750cbe21e62c72e1f7a858fe0ec83007fccd0d6836018a531b97766edba1ab\": container with ID starting with 4c750cbe21e62c72e1f7a858fe0ec83007fccd0d6836018a531b97766edba1ab not found: ID does not exist" containerID="4c750cbe21e62c72e1f7a858fe0ec83007fccd0d6836018a531b97766edba1ab" Jul 26 16:38:52.254167 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:52.254132 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c750cbe21e62c72e1f7a858fe0ec83007fccd0d6836018a531b97766edba1ab"} err="failed to get container status \"4c750cbe21e62c72e1f7a858fe0ec83007fccd0d6836018a531b97766edba1ab\": rpc error: code = NotFound desc = could not find container \"4c750cbe21e62c72e1f7a858fe0ec83007fccd0d6836018a531b97766edba1ab\": container with ID starting with 4c750cbe21e62c72e1f7a858fe0ec83007fccd0d6836018a531b97766edba1ab not found: ID does not exist" Jul 26 16:38:52.266164 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:52.266126 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr"] Jul 26 16:38:52.266326 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:52.266179 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2774653d-openshift-default-5497c4d696-hj6tr"] Jul 26 16:38:52.890604 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:38:52.890559 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b967e2cc-9c99-4654-88ce-ee42fb19fe99" path="/var/lib/kubelet/pods/b967e2cc-9c99-4654-88ce-ee42fb19fe99/volumes" Jul 26 16:40:24.967655 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.967579 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh"] Jul 26 16:40:24.968135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.967921 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" containerName="istio-proxy" Jul 26 16:40:24.968135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.967933 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" containerName="istio-proxy" Jul 26 16:40:24.968135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.967942 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" containerName="istio-proxy" Jul 26 16:40:24.968135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.967947 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" containerName="istio-proxy" Jul 26 16:40:24.968135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.967957 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b967e2cc-9c99-4654-88ce-ee42fb19fe99" containerName="istio-proxy" Jul 26 16:40:24.968135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.967963 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b967e2cc-9c99-4654-88ce-ee42fb19fe99" containerName="istio-proxy" Jul 26 16:40:24.968135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.967972 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9ce8931-d65c-4247-9475-ade8884e9db4" containerName="maas-api" Jul 26 16:40:24.968135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.967978 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9ce8931-d65c-4247-9475-ade8884e9db4" containerName="maas-api" Jul 26 16:40:24.968135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.967985 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b967e2cc-9c99-4654-88ce-ee42fb19fe99" containerName="istio-proxy" Jul 26 16:40:24.968135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.967990 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b967e2cc-9c99-4654-88ce-ee42fb19fe99" containerName="istio-proxy" Jul 26 16:40:24.968135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.968050 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" containerName="istio-proxy" Jul 26 16:40:24.968135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.968061 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf6e5410-904c-43ca-b3d0-dcd7b30a82bc" containerName="istio-proxy" Jul 26 16:40:24.968135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.968068 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b967e2cc-9c99-4654-88ce-ee42fb19fe99" containerName="istio-proxy" Jul 26 16:40:24.968135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.968075 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9ce8931-d65c-4247-9475-ade8884e9db4" containerName="maas-api" Jul 26 16:40:24.968135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.968080 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b967e2cc-9c99-4654-88ce-ee42fb19fe99" containerName="istio-proxy" Jul 26 16:40:24.971161 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.971138 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:24.973940 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.973918 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d2caf295-openshift-default-dockercfg-shv5w\"" Jul 26 16:40:24.992238 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:24.992188 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh"] Jul 26 16:40:25.096356 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.096313 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcdhq\" (UniqueName: \"kubernetes.io/projected/78125794-1388-4037-b21e-7f1d532d3651-kube-api-access-xcdhq\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.096356 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.096355 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-workload-socket\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.096567 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.096377 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78125794-1388-4037-b21e-7f1d532d3651-istio-podinfo\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.096567 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.096469 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-istio-envoy\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.096567 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.096519 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-workload-certs\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.096671 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.096588 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-istio-data\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.096671 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.096623 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78125794-1388-4037-b21e-7f1d532d3651-istiod-ca-cert\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.096671 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.096645 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-credential-socket\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.096671 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.096665 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78125794-1388-4037-b21e-7f1d532d3651-istio-token\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.197960 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.197925 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78125794-1388-4037-b21e-7f1d532d3651-istio-podinfo\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.198120 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.197980 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-istio-envoy\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.198120 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.198021 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-workload-certs\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.198120 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.198069 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-istio-data\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.198120 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.198097 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78125794-1388-4037-b21e-7f1d532d3651-istiod-ca-cert\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.198374 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.198130 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-credential-socket\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.198374 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.198159 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78125794-1388-4037-b21e-7f1d532d3651-istio-token\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.198374 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.198195 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcdhq\" (UniqueName: \"kubernetes.io/projected/78125794-1388-4037-b21e-7f1d532d3651-kube-api-access-xcdhq\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.198374 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.198243 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-workload-socket\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.198592 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.198514 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-workload-certs\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.198592 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.198530 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-istio-data\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.198894 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.198869 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-workload-socket\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.198894 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.198884 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78125794-1388-4037-b21e-7f1d532d3651-istiod-ca-cert\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.199016 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.198923 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-credential-socket\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.200321 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.200295 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-istio-envoy\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.200886 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.200859 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78125794-1388-4037-b21e-7f1d532d3651-istio-podinfo\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.206549 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.206527 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78125794-1388-4037-b21e-7f1d532d3651-istio-token\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.206793 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.206773 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcdhq\" (UniqueName: \"kubernetes.io/projected/78125794-1388-4037-b21e-7f1d532d3651-kube-api-access-xcdhq\") pod \"e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.281083 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.281047 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:25.419886 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.419854 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh"] Jul 26 16:40:25.421729 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:40:25.421704 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78125794_1388_4037_b21e_7f1d532d3651.slice/crio-b9511db19004169b55e0da0d36d876b3cd540e15104679ffe4c619a5c1f86e9e WatchSource:0}: Error finding container b9511db19004169b55e0da0d36d876b3cd540e15104679ffe4c619a5c1f86e9e: Status 404 returned error can't find the container with id b9511db19004169b55e0da0d36d876b3cd540e15104679ffe4c619a5c1f86e9e Jul 26 16:40:25.423803 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.423773 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:40:25.423907 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.423846 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:40:25.423907 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.423893 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:40:25.745343 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.745257 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" event={"ID":"78125794-1388-4037-b21e-7f1d532d3651","Type":"ContainerStarted","Data":"a77c84e9838ade166a2f26386011f48e65b91abed80486deb91554543f3ad708"} Jul 26 16:40:25.745343 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.745325 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" event={"ID":"78125794-1388-4037-b21e-7f1d532d3651","Type":"ContainerStarted","Data":"b9511db19004169b55e0da0d36d876b3cd540e15104679ffe4c619a5c1f86e9e"} Jul 26 16:40:25.766359 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:25.766313 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" podStartSLOduration=1.76629792 podStartE2EDuration="1.76629792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:40:25.763612715 +0000 UTC m=+1899.344699285" watchObservedRunningTime="2026-07-26 16:40:25.76629792 +0000 UTC m=+1899.347384489" Jul 26 16:40:26.281330 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:26.281275 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:26.286313 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:26.286289 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:26.750337 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:26.750249 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:26.751190 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:26.751171 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:27.758536 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:27.758502 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj"] Jul 26 16:40:27.761963 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:27.761943 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" Jul 26 16:40:27.764709 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:27.764666 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d2caf295-serving-cert\"" Jul 26 16:40:27.774013 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:27.773985 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj"] Jul 26 16:40:27.820346 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:27.820312 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc2df7f-fa60-46ca-917d-286bf12f8017-openshift-service-ca\") pod \"maas-api-e2e-mt-d2caf295-f58cdc689-74cbj\" (UID: \"5dc2df7f-fa60-46ca-917d-286bf12f8017\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" Jul 26 16:40:27.820346 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:27.820349 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc2df7f-fa60-46ca-917d-286bf12f8017-maas-api-tls\") pod \"maas-api-e2e-mt-d2caf295-f58cdc689-74cbj\" (UID: \"5dc2df7f-fa60-46ca-917d-286bf12f8017\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" Jul 26 16:40:27.820546 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:27.820374 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq6pd\" (UniqueName: \"kubernetes.io/projected/5dc2df7f-fa60-46ca-917d-286bf12f8017-kube-api-access-mq6pd\") pod \"maas-api-e2e-mt-d2caf295-f58cdc689-74cbj\" (UID: \"5dc2df7f-fa60-46ca-917d-286bf12f8017\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" Jul 26 16:40:27.921351 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:27.921309 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq6pd\" (UniqueName: \"kubernetes.io/projected/5dc2df7f-fa60-46ca-917d-286bf12f8017-kube-api-access-mq6pd\") pod \"maas-api-e2e-mt-d2caf295-f58cdc689-74cbj\" (UID: \"5dc2df7f-fa60-46ca-917d-286bf12f8017\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" Jul 26 16:40:27.921541 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:27.921416 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc2df7f-fa60-46ca-917d-286bf12f8017-openshift-service-ca\") pod \"maas-api-e2e-mt-d2caf295-f58cdc689-74cbj\" (UID: \"5dc2df7f-fa60-46ca-917d-286bf12f8017\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" Jul 26 16:40:27.921541 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:27.921433 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc2df7f-fa60-46ca-917d-286bf12f8017-maas-api-tls\") pod \"maas-api-e2e-mt-d2caf295-f58cdc689-74cbj\" (UID: \"5dc2df7f-fa60-46ca-917d-286bf12f8017\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" Jul 26 16:40:27.921541 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:40:27.921531 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-serving-cert: secret "maas-api-e2e-mt-d2caf295-serving-cert" not found Jul 26 16:40:27.921652 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:40:27.921595 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dc2df7f-fa60-46ca-917d-286bf12f8017-maas-api-tls podName:5dc2df7f-fa60-46ca-917d-286bf12f8017 nodeName:}" failed. No retries permitted until 2026-07-26 16:40:28.421578105 +0000 UTC m=+1902.002664653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5dc2df7f-fa60-46ca-917d-286bf12f8017-maas-api-tls") pod "maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" (UID: "5dc2df7f-fa60-46ca-917d-286bf12f8017") : secret "maas-api-e2e-mt-d2caf295-serving-cert" not found Jul 26 16:40:27.922101 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:27.922078 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc2df7f-fa60-46ca-917d-286bf12f8017-openshift-service-ca\") pod \"maas-api-e2e-mt-d2caf295-f58cdc689-74cbj\" (UID: \"5dc2df7f-fa60-46ca-917d-286bf12f8017\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" Jul 26 16:40:27.931046 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:27.931010 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq6pd\" (UniqueName: \"kubernetes.io/projected/5dc2df7f-fa60-46ca-917d-286bf12f8017-kube-api-access-mq6pd\") pod \"maas-api-e2e-mt-d2caf295-f58cdc689-74cbj\" (UID: \"5dc2df7f-fa60-46ca-917d-286bf12f8017\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" Jul 26 16:40:28.427644 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:28.427601 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc2df7f-fa60-46ca-917d-286bf12f8017-maas-api-tls\") pod \"maas-api-e2e-mt-d2caf295-f58cdc689-74cbj\" (UID: \"5dc2df7f-fa60-46ca-917d-286bf12f8017\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" Jul 26 16:40:28.430121 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:28.430096 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc2df7f-fa60-46ca-917d-286bf12f8017-maas-api-tls\") pod \"maas-api-e2e-mt-d2caf295-f58cdc689-74cbj\" (UID: \"5dc2df7f-fa60-46ca-917d-286bf12f8017\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" Jul 26 16:40:28.674378 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:28.674335 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" Jul 26 16:40:28.821964 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:28.821936 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj"] Jul 26 16:40:28.823513 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:40:28.823487 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc2df7f_fa60_46ca_917d_286bf12f8017.slice/crio-04ab9cc42c51c7bbeff4508ca112f717a2de8c5c414e6e84a3e21e5542cb4093 WatchSource:0}: Error finding container 04ab9cc42c51c7bbeff4508ca112f717a2de8c5c414e6e84a3e21e5542cb4093: Status 404 returned error can't find the container with id 04ab9cc42c51c7bbeff4508ca112f717a2de8c5c414e6e84a3e21e5542cb4093 Jul 26 16:40:29.771444 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:29.771340 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" event={"ID":"5dc2df7f-fa60-46ca-917d-286bf12f8017","Type":"ContainerStarted","Data":"45b302066e4d4fdf2a1665cecfde531232e9f1c74f9277b7327c200a5abfae47"} Jul 26 16:40:29.771444 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:29.771406 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" event={"ID":"5dc2df7f-fa60-46ca-917d-286bf12f8017","Type":"ContainerStarted","Data":"04ab9cc42c51c7bbeff4508ca112f717a2de8c5c414e6e84a3e21e5542cb4093"} Jul 26 16:40:29.771684 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:29.771498 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" Jul 26 16:40:29.795254 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:29.795189 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" podStartSLOduration=2.235276701 podStartE2EDuration="2.795172821s" totalImagesPullingTime="559.89612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:40:29.792031953 +0000 UTC m=+1903.373118524" watchObservedRunningTime="2026-07-26 16:40:29.795172821 +0000 UTC m=+1903.376259419" Jul 26 16:40:35.781423 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:35.781392 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" Jul 26 16:40:39.616234 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:39.616181 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2caf295-4wjgp"] Jul 26 16:40:39.620006 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:39.619989 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2caf295-4wjgp" Jul 26 16:40:39.623409 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:39.623386 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2xlcn\"" Jul 26 16:40:39.627940 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:39.627915 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2caf295-4wjgp"] Jul 26 16:40:39.732313 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:39.732273 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d2caf295-4wjgp\" (UID: \"6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2caf295-4wjgp" Jul 26 16:40:39.833636 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:39.833598 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d2caf295-4wjgp\" (UID: \"6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2caf295-4wjgp" Jul 26 16:40:39.834358 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:39.834335 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d2caf295-4wjgp\" (UID: \"6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2caf295-4wjgp" Jul 26 16:40:39.932643 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:39.932557 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2caf295-4wjgp" Jul 26 16:40:40.108598 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:40.108567 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2caf295-4wjgp"] Jul 26 16:40:40.110385 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:40:40.110355 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd6ccc6_aae4_4630_9eb6_8aa675a12c1c.slice/crio-451cb5a22bc2f54033359d23100ae88dbf675c7832374f5fb0629deed9159f2e WatchSource:0}: Error finding container 451cb5a22bc2f54033359d23100ae88dbf675c7832374f5fb0629deed9159f2e: Status 404 returned error can't find the container with id 451cb5a22bc2f54033359d23100ae88dbf675c7832374f5fb0629deed9159f2e Jul 26 16:40:40.830812 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:40.830770 2563 generic.go:356] "Generic (PLEG): container finished" podID="6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c" containerID="e6229ee9225b83181ea3e26d059426c85d435922180c8702aed3ec77800bf83f" exitCode=0 Jul 26 16:40:40.831273 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:40.830848 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2caf295-4wjgp" event={"ID":"6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c","Type":"ContainerDied","Data":"e6229ee9225b83181ea3e26d059426c85d435922180c8702aed3ec77800bf83f"} Jul 26 16:40:40.831273 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:40.830932 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2caf295-4wjgp" event={"ID":"6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c","Type":"ContainerStarted","Data":"451cb5a22bc2f54033359d23100ae88dbf675c7832374f5fb0629deed9159f2e"} Jul 26 16:40:41.970493 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:41.970467 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2caf295-4wjgp" Jul 26 16:40:42.055844 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:42.055811 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c-maas-api-service-ca\") pod \"6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c\" (UID: \"6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c\") " Jul 26 16:40:42.056174 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:42.056151 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c" (UID: "6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:40:42.156570 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:42.156482 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c-maas-api-service-ca\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:40:42.791457 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:42.791405 2563 patch_prober.go:28] interesting pod/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 26 16:40:42.791666 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:42.791476 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" podUID="78125794-1388-4037-b21e-7f1d532d3651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 26 16:40:42.844171 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:42.844139 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2caf295-4wjgp" Jul 26 16:40:42.846414 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:42.844139 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2caf295-4wjgp" event={"ID":"6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c","Type":"ContainerDied","Data":"451cb5a22bc2f54033359d23100ae88dbf675c7832374f5fb0629deed9159f2e"} Jul 26 16:40:42.846414 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:42.844284 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="451cb5a22bc2f54033359d23100ae88dbf675c7832374f5fb0629deed9159f2e" Jul 26 16:40:43.850881 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:43.850843 2563 generic.go:356] "Generic (PLEG): container finished" podID="78125794-1388-4037-b21e-7f1d532d3651" containerID="a77c84e9838ade166a2f26386011f48e65b91abed80486deb91554543f3ad708" exitCode=0 Jul 26 16:40:43.851433 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:43.850919 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" event={"ID":"78125794-1388-4037-b21e-7f1d532d3651","Type":"ContainerDied","Data":"a77c84e9838ade166a2f26386011f48e65b91abed80486deb91554543f3ad708"} Jul 26 16:40:43.851505 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:43.851447 2563 scope.go:117] "RemoveContainer" containerID="a77c84e9838ade166a2f26386011f48e65b91abed80486deb91554543f3ad708" Jul 26 16:40:43.852488 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:43.852463 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:40:43.852558 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:43.852532 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:40:43.852596 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:43.852570 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:40:44.858350 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:44.858263 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" event={"ID":"78125794-1388-4037-b21e-7f1d532d3651","Type":"ContainerStarted","Data":"2ca4f6e9f23347a6803b194bf8bb67ee2d9796bd8eb0877da790810fb8bea0f4"} Jul 26 16:40:45.281328 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:45.281285 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:45.281679 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:45.281648 2563 patch_prober.go:28] interesting pod/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 26 16:40:45.281787 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:45.281714 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" podUID="78125794-1388-4037-b21e-7f1d532d3651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 26 16:40:46.282563 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:46.282521 2563 patch_prober.go:28] interesting pod/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 26 16:40:46.282964 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:46.282584 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" podUID="78125794-1388-4037-b21e-7f1d532d3651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 26 16:40:47.282423 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:47.282382 2563 patch_prober.go:28] interesting pod/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 26 16:40:47.282647 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:47.282455 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" podUID="78125794-1388-4037-b21e-7f1d532d3651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 26 16:40:48.282054 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:48.282018 2563 patch_prober.go:28] interesting pod/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 26 16:40:48.282335 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:48.282295 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" podUID="78125794-1388-4037-b21e-7f1d532d3651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 26 16:40:49.285156 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:49.285123 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:49.285552 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:49.285520 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:49.286293 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:49.286271 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:40:50.083429 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.083393 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj"] Jul 26 16:40:50.083626 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.083580 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" podUID="5dc2df7f-fa60-46ca-917d-286bf12f8017" containerName="maas-api" containerID="cri-o://45b302066e4d4fdf2a1665cecfde531232e9f1c74f9277b7327c200a5abfae47" gracePeriod=30 Jul 26 16:40:50.341548 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.341471 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" Jul 26 16:40:50.434295 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.434255 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq6pd\" (UniqueName: \"kubernetes.io/projected/5dc2df7f-fa60-46ca-917d-286bf12f8017-kube-api-access-mq6pd\") pod \"5dc2df7f-fa60-46ca-917d-286bf12f8017\" (UID: \"5dc2df7f-fa60-46ca-917d-286bf12f8017\") " Jul 26 16:40:50.434490 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.434333 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc2df7f-fa60-46ca-917d-286bf12f8017-openshift-service-ca\") pod \"5dc2df7f-fa60-46ca-917d-286bf12f8017\" (UID: \"5dc2df7f-fa60-46ca-917d-286bf12f8017\") " Jul 26 16:40:50.434490 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.434381 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc2df7f-fa60-46ca-917d-286bf12f8017-maas-api-tls\") pod \"5dc2df7f-fa60-46ca-917d-286bf12f8017\" (UID: \"5dc2df7f-fa60-46ca-917d-286bf12f8017\") " Jul 26 16:40:50.434745 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.434718 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc2df7f-fa60-46ca-917d-286bf12f8017-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5dc2df7f-fa60-46ca-917d-286bf12f8017" (UID: "5dc2df7f-fa60-46ca-917d-286bf12f8017"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:40:50.436356 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.436331 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dc2df7f-fa60-46ca-917d-286bf12f8017-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5dc2df7f-fa60-46ca-917d-286bf12f8017" (UID: "5dc2df7f-fa60-46ca-917d-286bf12f8017"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:40:50.436463 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.436394 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc2df7f-fa60-46ca-917d-286bf12f8017-kube-api-access-mq6pd" (OuterVolumeSpecName: "kube-api-access-mq6pd") pod "5dc2df7f-fa60-46ca-917d-286bf12f8017" (UID: "5dc2df7f-fa60-46ca-917d-286bf12f8017"). InnerVolumeSpecName "kube-api-access-mq6pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:40:50.535907 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.535867 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc2df7f-fa60-46ca-917d-286bf12f8017-maas-api-tls\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:40:50.535907 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.535903 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq6pd\" (UniqueName: \"kubernetes.io/projected/5dc2df7f-fa60-46ca-917d-286bf12f8017-kube-api-access-mq6pd\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:40:50.535907 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.535914 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc2df7f-fa60-46ca-917d-286bf12f8017-openshift-service-ca\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:40:50.899760 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.899728 2563 generic.go:356] "Generic (PLEG): container finished" podID="5dc2df7f-fa60-46ca-917d-286bf12f8017" containerID="45b302066e4d4fdf2a1665cecfde531232e9f1c74f9277b7327c200a5abfae47" exitCode=0 Jul 26 16:40:50.899885 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.899785 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" Jul 26 16:40:50.899885 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.899807 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" event={"ID":"5dc2df7f-fa60-46ca-917d-286bf12f8017","Type":"ContainerDied","Data":"45b302066e4d4fdf2a1665cecfde531232e9f1c74f9277b7327c200a5abfae47"} Jul 26 16:40:50.899885 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.899855 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj" event={"ID":"5dc2df7f-fa60-46ca-917d-286bf12f8017","Type":"ContainerDied","Data":"04ab9cc42c51c7bbeff4508ca112f717a2de8c5c414e6e84a3e21e5542cb4093"} Jul 26 16:40:50.899885 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.899878 2563 scope.go:117] "RemoveContainer" containerID="45b302066e4d4fdf2a1665cecfde531232e9f1c74f9277b7327c200a5abfae47" Jul 26 16:40:50.908532 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.908513 2563 scope.go:117] "RemoveContainer" containerID="45b302066e4d4fdf2a1665cecfde531232e9f1c74f9277b7327c200a5abfae47" Jul 26 16:40:50.908780 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:40:50.908762 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b302066e4d4fdf2a1665cecfde531232e9f1c74f9277b7327c200a5abfae47\": container with ID starting with 45b302066e4d4fdf2a1665cecfde531232e9f1c74f9277b7327c200a5abfae47 not found: ID does not exist" containerID="45b302066e4d4fdf2a1665cecfde531232e9f1c74f9277b7327c200a5abfae47" Jul 26 16:40:50.908817 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.908789 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b302066e4d4fdf2a1665cecfde531232e9f1c74f9277b7327c200a5abfae47"} err="failed to get container status \"45b302066e4d4fdf2a1665cecfde531232e9f1c74f9277b7327c200a5abfae47\": rpc error: code = NotFound desc = could not find container \"45b302066e4d4fdf2a1665cecfde531232e9f1c74f9277b7327c200a5abfae47\": container with ID starting with 45b302066e4d4fdf2a1665cecfde531232e9f1c74f9277b7327c200a5abfae47 not found: ID does not exist" Jul 26 16:40:50.918144 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.918116 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj"] Jul 26 16:40:50.924339 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:50.924316 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d2caf295-f58cdc689-74cbj"] Jul 26 16:40:52.890729 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:52.890692 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dc2df7f-fa60-46ca-917d-286bf12f8017" path="/var/lib/kubelet/pods/5dc2df7f-fa60-46ca-917d-286bf12f8017/volumes" Jul 26 16:40:54.687419 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:54.687373 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2caf295-4wjgp"] Jul 26 16:40:54.689976 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:54.689951 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2caf295-4wjgp"] Jul 26 16:40:54.890963 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:40:54.890927 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c" path="/var/lib/kubelet/pods/6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c/volumes" Jul 26 16:41:01.143020 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:01.142984 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh"] Jul 26 16:41:01.143624 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:01.143257 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" podUID="78125794-1388-4037-b21e-7f1d532d3651" containerName="istio-proxy" containerID="cri-o://2ca4f6e9f23347a6803b194bf8bb67ee2d9796bd8eb0877da790810fb8bea0f4" gracePeriod=30 Jul 26 16:41:04.287676 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:04.287640 2563 patch_prober.go:28] interesting pod/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 16:41:04.288046 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:04.287698 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" podUID="78125794-1388-4037-b21e-7f1d532d3651" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 16:41:06.399888 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.399864 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:41:06.581558 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.581521 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78125794-1388-4037-b21e-7f1d532d3651-istio-token\") pod \"78125794-1388-4037-b21e-7f1d532d3651\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " Jul 26 16:41:06.581750 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.581564 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-credential-socket\") pod \"78125794-1388-4037-b21e-7f1d532d3651\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " Jul 26 16:41:06.581750 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.581623 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcdhq\" (UniqueName: \"kubernetes.io/projected/78125794-1388-4037-b21e-7f1d532d3651-kube-api-access-xcdhq\") pod \"78125794-1388-4037-b21e-7f1d532d3651\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " Jul 26 16:41:06.581750 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.581660 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-workload-socket\") pod \"78125794-1388-4037-b21e-7f1d532d3651\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " Jul 26 16:41:06.581750 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.581681 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-workload-certs\") pod \"78125794-1388-4037-b21e-7f1d532d3651\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " Jul 26 16:41:06.581750 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.581707 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78125794-1388-4037-b21e-7f1d532d3651-istiod-ca-cert\") pod \"78125794-1388-4037-b21e-7f1d532d3651\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " Jul 26 16:41:06.582016 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.581800 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-istio-envoy\") pod \"78125794-1388-4037-b21e-7f1d532d3651\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " Jul 26 16:41:06.582016 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.581827 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78125794-1388-4037-b21e-7f1d532d3651-istio-podinfo\") pod \"78125794-1388-4037-b21e-7f1d532d3651\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " Jul 26 16:41:06.582016 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.581868 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-istio-data\") pod \"78125794-1388-4037-b21e-7f1d532d3651\" (UID: \"78125794-1388-4037-b21e-7f1d532d3651\") " Jul 26 16:41:06.582016 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.581953 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "78125794-1388-4037-b21e-7f1d532d3651" (UID: "78125794-1388-4037-b21e-7f1d532d3651"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:41:06.582290 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.581987 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "78125794-1388-4037-b21e-7f1d532d3651" (UID: "78125794-1388-4037-b21e-7f1d532d3651"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:41:06.582290 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.581981 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "78125794-1388-4037-b21e-7f1d532d3651" (UID: "78125794-1388-4037-b21e-7f1d532d3651"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:41:06.582290 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.582082 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78125794-1388-4037-b21e-7f1d532d3651-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "78125794-1388-4037-b21e-7f1d532d3651" (UID: "78125794-1388-4037-b21e-7f1d532d3651"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:41:06.582290 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.582240 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-credential-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:41:06.582290 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.582260 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-workload-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:41:06.582290 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.582275 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-workload-certs\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:41:06.582290 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.582289 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78125794-1388-4037-b21e-7f1d532d3651-istiod-ca-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:41:06.582851 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.582829 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-istio-data" (OuterVolumeSpecName: "istio-data") pod "78125794-1388-4037-b21e-7f1d532d3651" (UID: "78125794-1388-4037-b21e-7f1d532d3651"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:41:06.583853 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.583829 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78125794-1388-4037-b21e-7f1d532d3651-istio-token" (OuterVolumeSpecName: "istio-token") pod "78125794-1388-4037-b21e-7f1d532d3651" (UID: "78125794-1388-4037-b21e-7f1d532d3651"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:41:06.583942 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.583924 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78125794-1388-4037-b21e-7f1d532d3651-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "78125794-1388-4037-b21e-7f1d532d3651" (UID: "78125794-1388-4037-b21e-7f1d532d3651"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:41:06.584414 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.584397 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78125794-1388-4037-b21e-7f1d532d3651-kube-api-access-xcdhq" (OuterVolumeSpecName: "kube-api-access-xcdhq") pod "78125794-1388-4037-b21e-7f1d532d3651" (UID: "78125794-1388-4037-b21e-7f1d532d3651"). InnerVolumeSpecName "kube-api-access-xcdhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:41:06.584414 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.584399 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "78125794-1388-4037-b21e-7f1d532d3651" (UID: "78125794-1388-4037-b21e-7f1d532d3651"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:41:06.683317 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.683282 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcdhq\" (UniqueName: \"kubernetes.io/projected/78125794-1388-4037-b21e-7f1d532d3651-kube-api-access-xcdhq\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:41:06.683317 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.683313 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-istio-envoy\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:41:06.683317 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.683324 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78125794-1388-4037-b21e-7f1d532d3651-istio-podinfo\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:41:06.683551 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.683333 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78125794-1388-4037-b21e-7f1d532d3651-istio-data\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:41:06.683551 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.683341 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78125794-1388-4037-b21e-7f1d532d3651-istio-token\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:41:06.986352 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.986316 2563 generic.go:356] "Generic (PLEG): container finished" podID="78125794-1388-4037-b21e-7f1d532d3651" containerID="2ca4f6e9f23347a6803b194bf8bb67ee2d9796bd8eb0877da790810fb8bea0f4" exitCode=0 Jul 26 16:41:06.986499 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.986361 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" Jul 26 16:41:06.986499 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.986365 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" event={"ID":"78125794-1388-4037-b21e-7f1d532d3651","Type":"ContainerDied","Data":"2ca4f6e9f23347a6803b194bf8bb67ee2d9796bd8eb0877da790810fb8bea0f4"} Jul 26 16:41:06.986499 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.986415 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh" event={"ID":"78125794-1388-4037-b21e-7f1d532d3651","Type":"ContainerDied","Data":"b9511db19004169b55e0da0d36d876b3cd540e15104679ffe4c619a5c1f86e9e"} Jul 26 16:41:06.986499 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.986437 2563 scope.go:117] "RemoveContainer" containerID="2ca4f6e9f23347a6803b194bf8bb67ee2d9796bd8eb0877da790810fb8bea0f4" Jul 26 16:41:06.995521 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:06.995505 2563 scope.go:117] "RemoveContainer" containerID="a77c84e9838ade166a2f26386011f48e65b91abed80486deb91554543f3ad708" Jul 26 16:41:07.004052 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:07.004026 2563 scope.go:117] "RemoveContainer" containerID="2ca4f6e9f23347a6803b194bf8bb67ee2d9796bd8eb0877da790810fb8bea0f4" Jul 26 16:41:07.004412 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:41:07.004381 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca4f6e9f23347a6803b194bf8bb67ee2d9796bd8eb0877da790810fb8bea0f4\": container with ID starting with 2ca4f6e9f23347a6803b194bf8bb67ee2d9796bd8eb0877da790810fb8bea0f4 not found: ID does not exist" containerID="2ca4f6e9f23347a6803b194bf8bb67ee2d9796bd8eb0877da790810fb8bea0f4" Jul 26 16:41:07.004501 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:07.004418 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca4f6e9f23347a6803b194bf8bb67ee2d9796bd8eb0877da790810fb8bea0f4"} err="failed to get container status \"2ca4f6e9f23347a6803b194bf8bb67ee2d9796bd8eb0877da790810fb8bea0f4\": rpc error: code = NotFound desc = could not find container \"2ca4f6e9f23347a6803b194bf8bb67ee2d9796bd8eb0877da790810fb8bea0f4\": container with ID starting with 2ca4f6e9f23347a6803b194bf8bb67ee2d9796bd8eb0877da790810fb8bea0f4 not found: ID does not exist" Jul 26 16:41:07.004501 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:07.004437 2563 scope.go:117] "RemoveContainer" containerID="a77c84e9838ade166a2f26386011f48e65b91abed80486deb91554543f3ad708" Jul 26 16:41:07.004724 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:41:07.004706 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77c84e9838ade166a2f26386011f48e65b91abed80486deb91554543f3ad708\": container with ID starting with a77c84e9838ade166a2f26386011f48e65b91abed80486deb91554543f3ad708 not found: ID does not exist" containerID="a77c84e9838ade166a2f26386011f48e65b91abed80486deb91554543f3ad708" Jul 26 16:41:07.004772 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:07.004732 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77c84e9838ade166a2f26386011f48e65b91abed80486deb91554543f3ad708"} err="failed to get container status \"a77c84e9838ade166a2f26386011f48e65b91abed80486deb91554543f3ad708\": rpc error: code = NotFound desc = could not find container \"a77c84e9838ade166a2f26386011f48e65b91abed80486deb91554543f3ad708\": container with ID starting with a77c84e9838ade166a2f26386011f48e65b91abed80486deb91554543f3ad708 not found: ID does not exist" Jul 26 16:41:07.005615 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:07.005595 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh"] Jul 26 16:41:07.009129 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:07.009103 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d2caf295-openshift-default-5d88cc86d-msqjh"] Jul 26 16:41:08.890660 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:08.890627 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78125794-1388-4037-b21e-7f1d532d3651" path="/var/lib/kubelet/pods/78125794-1388-4037-b21e-7f1d532d3651/volumes" Jul 26 16:41:21.019702 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.019661 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q"] Jul 26 16:41:21.020316 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.020164 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78125794-1388-4037-b21e-7f1d532d3651" containerName="istio-proxy" Jul 26 16:41:21.020316 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.020185 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="78125794-1388-4037-b21e-7f1d532d3651" containerName="istio-proxy" Jul 26 16:41:21.020316 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.020202 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c" containerName="revoke-keys" Jul 26 16:41:21.020316 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.020211 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c" containerName="revoke-keys" Jul 26 16:41:21.020316 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.020250 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc2df7f-fa60-46ca-917d-286bf12f8017" containerName="maas-api" Jul 26 16:41:21.020316 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.020261 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc2df7f-fa60-46ca-917d-286bf12f8017" containerName="maas-api" Jul 26 16:41:21.020640 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.020387 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="78125794-1388-4037-b21e-7f1d532d3651" containerName="istio-proxy" Jul 26 16:41:21.020640 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.020401 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd6ccc6-aae4-4630-9eb6-8aa675a12c1c" containerName="revoke-keys" Jul 26 16:41:21.020640 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.020412 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dc2df7f-fa60-46ca-917d-286bf12f8017" containerName="maas-api" Jul 26 16:41:21.023864 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.023841 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.026600 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.026576 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ffa73817-openshift-default-dockercfg-ms4v4\"" Jul 26 16:41:21.037672 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.037647 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q"] Jul 26 16:41:21.106955 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.106915 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-token\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.107132 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.106963 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-workload-socket\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.107132 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.106998 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-workload-certs\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.107132 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.107043 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-podinfo\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.107303 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.107134 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-envoy\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.107303 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.107162 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-data\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.107303 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.107189 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-credential-socket\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.107303 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.107209 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istiod-ca-cert\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.107434 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.107309 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmrb8\" (UniqueName: \"kubernetes.io/projected/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-kube-api-access-jmrb8\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.208672 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.208629 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-workload-socket\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.208880 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.208692 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-workload-certs\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.208880 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.208729 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-podinfo\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.208880 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.208808 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-envoy\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.208880 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.208831 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-data\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.208880 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.208861 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-credential-socket\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.208880 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.208882 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istiod-ca-cert\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.209170 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.208918 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmrb8\" (UniqueName: \"kubernetes.io/projected/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-kube-api-access-jmrb8\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.209170 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.208973 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-token\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.209170 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.209160 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-workload-socket\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.209510 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.209477 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-workload-certs\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.209642 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.209539 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-data\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.209642 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.209570 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-credential-socket\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.209949 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.209926 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istiod-ca-cert\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.211158 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.211136 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-envoy\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.211321 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.211295 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-podinfo\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.219039 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.219014 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmrb8\" (UniqueName: \"kubernetes.io/projected/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-kube-api-access-jmrb8\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.222503 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.222479 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-token\") pod \"e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.334062 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.334020 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:21.469827 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.469798 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q"] Jul 26 16:41:21.471823 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:41:21.471787 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f061ec_6ce6_4e9d_a0c2_ec564f607756.slice/crio-29bcec8867592b7e8f9d2d0fd418a133c808e98b5d82bf63d395ca18a4b8dc37 WatchSource:0}: Error finding container 29bcec8867592b7e8f9d2d0fd418a133c808e98b5d82bf63d395ca18a4b8dc37: Status 404 returned error can't find the container with id 29bcec8867592b7e8f9d2d0fd418a133c808e98b5d82bf63d395ca18a4b8dc37 Jul 26 16:41:21.473867 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.473837 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:41:21.473944 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.473902 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:41:21.473944 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:21.473932 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:41:22.070609 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:22.070572 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" event={"ID":"e5f061ec-6ce6-4e9d-a0c2-ec564f607756","Type":"ContainerStarted","Data":"95fa8497ac4d87d7e56d98adcf782ee19dd893f9d12a5f8ce0301d229cb1679a"} Jul 26 16:41:22.071011 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:22.070631 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" event={"ID":"e5f061ec-6ce6-4e9d-a0c2-ec564f607756","Type":"ContainerStarted","Data":"29bcec8867592b7e8f9d2d0fd418a133c808e98b5d82bf63d395ca18a4b8dc37"} Jul 26 16:41:22.099595 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:22.099542 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" podStartSLOduration=2.099525822 podStartE2EDuration="2.099525822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:41:22.097117673 +0000 UTC m=+1955.678204239" watchObservedRunningTime="2026-07-26 16:41:22.099525822 +0000 UTC m=+1955.680612391" Jul 26 16:41:22.334663 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:22.334562 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:22.339792 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:22.339766 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:23.076775 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:23.076744 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:23.077642 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:23.077620 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:24.143657 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.143626 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c"] Jul 26 16:41:24.144031 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.143986 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78125794-1388-4037-b21e-7f1d532d3651" containerName="istio-proxy" Jul 26 16:41:24.144031 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.143998 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="78125794-1388-4037-b21e-7f1d532d3651" containerName="istio-proxy" Jul 26 16:41:24.144103 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.144054 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="78125794-1388-4037-b21e-7f1d532d3651" containerName="istio-proxy" Jul 26 16:41:24.147167 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.147148 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" Jul 26 16:41:24.150156 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.150134 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ffa73817-serving-cert\"" Jul 26 16:41:24.166034 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.166005 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c"] Jul 26 16:41:24.241960 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.241920 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvzxx\" (UniqueName: \"kubernetes.io/projected/ab9494e7-b55e-496c-8541-8554ce8fe83e-kube-api-access-rvzxx\") pod \"maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c\" (UID: \"ab9494e7-b55e-496c-8541-8554ce8fe83e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" Jul 26 16:41:24.242155 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.241971 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab9494e7-b55e-496c-8541-8554ce8fe83e-maas-api-tls\") pod \"maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c\" (UID: \"ab9494e7-b55e-496c-8541-8554ce8fe83e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" Jul 26 16:41:24.242155 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.242066 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab9494e7-b55e-496c-8541-8554ce8fe83e-openshift-service-ca\") pod \"maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c\" (UID: \"ab9494e7-b55e-496c-8541-8554ce8fe83e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" Jul 26 16:41:24.343434 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.343394 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab9494e7-b55e-496c-8541-8554ce8fe83e-openshift-service-ca\") pod \"maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c\" (UID: \"ab9494e7-b55e-496c-8541-8554ce8fe83e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" Jul 26 16:41:24.343637 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.343489 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvzxx\" (UniqueName: \"kubernetes.io/projected/ab9494e7-b55e-496c-8541-8554ce8fe83e-kube-api-access-rvzxx\") pod \"maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c\" (UID: \"ab9494e7-b55e-496c-8541-8554ce8fe83e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" Jul 26 16:41:24.343637 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.343517 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab9494e7-b55e-496c-8541-8554ce8fe83e-maas-api-tls\") pod \"maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c\" (UID: \"ab9494e7-b55e-496c-8541-8554ce8fe83e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" Jul 26 16:41:24.344110 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.344083 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab9494e7-b55e-496c-8541-8554ce8fe83e-openshift-service-ca\") pod \"maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c\" (UID: \"ab9494e7-b55e-496c-8541-8554ce8fe83e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" Jul 26 16:41:24.346055 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.346034 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab9494e7-b55e-496c-8541-8554ce8fe83e-maas-api-tls\") pod \"maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c\" (UID: \"ab9494e7-b55e-496c-8541-8554ce8fe83e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" Jul 26 16:41:24.360085 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.360059 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvzxx\" (UniqueName: \"kubernetes.io/projected/ab9494e7-b55e-496c-8541-8554ce8fe83e-kube-api-access-rvzxx\") pod \"maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c\" (UID: \"ab9494e7-b55e-496c-8541-8554ce8fe83e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" Jul 26 16:41:24.457889 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.457796 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" Jul 26 16:41:24.646652 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.646624 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c"] Jul 26 16:41:24.647975 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:41:24.647951 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9494e7_b55e_496c_8541_8554ce8fe83e.slice/crio-e08ac911cbb55ca5b1588c471a26e0c066bef2b7663bbbfa9fa635fb0aeb391b WatchSource:0}: Error finding container e08ac911cbb55ca5b1588c471a26e0c066bef2b7663bbbfa9fa635fb0aeb391b: Status 404 returned error can't find the container with id e08ac911cbb55ca5b1588c471a26e0c066bef2b7663bbbfa9fa635fb0aeb391b Jul 26 16:41:24.649374 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:24.649356 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 16:41:25.088263 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:25.088206 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" event={"ID":"ab9494e7-b55e-496c-8541-8554ce8fe83e","Type":"ContainerStarted","Data":"e08ac911cbb55ca5b1588c471a26e0c066bef2b7663bbbfa9fa635fb0aeb391b"} Jul 26 16:41:26.094977 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:26.094939 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" event={"ID":"ab9494e7-b55e-496c-8541-8554ce8fe83e","Type":"ContainerStarted","Data":"885292e9fc479bc9459f9041034ada28c6a1c0f8c41131af333ef16f5f46c9dc"} Jul 26 16:41:26.095391 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:26.095027 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" Jul 26 16:41:26.127613 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:26.127561 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" podStartSLOduration=1.720685917 podStartE2EDuration="2.127545072s" totalImagesPullingTime="406.859155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:41:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:41:26.125085102 +0000 UTC m=+1959.706171685" watchObservedRunningTime="2026-07-26 16:41:26.127545072 +0000 UTC m=+1959.708631702" Jul 26 16:41:32.105655 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:32.105621 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" Jul 26 16:41:38.337877 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:38.337371 2563 patch_prober.go:28] interesting pod/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:35498->10.134.0.55:15021: read: connection reset by peer" start-of-body= Jul 26 16:41:38.337877 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:38.337430 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" podUID="e5f061ec-6ce6-4e9d-a0c2-ec564f607756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:35498->10.134.0.55:15021: read: connection reset by peer" Jul 26 16:41:39.173469 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:39.173436 2563 generic.go:356] "Generic (PLEG): container finished" podID="e5f061ec-6ce6-4e9d-a0c2-ec564f607756" containerID="95fa8497ac4d87d7e56d98adcf782ee19dd893f9d12a5f8ce0301d229cb1679a" exitCode=0 Jul 26 16:41:39.173660 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:39.173525 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" event={"ID":"e5f061ec-6ce6-4e9d-a0c2-ec564f607756","Type":"ContainerDied","Data":"95fa8497ac4d87d7e56d98adcf782ee19dd893f9d12a5f8ce0301d229cb1679a"} Jul 26 16:41:39.173961 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:39.173947 2563 scope.go:117] "RemoveContainer" containerID="95fa8497ac4d87d7e56d98adcf782ee19dd893f9d12a5f8ce0301d229cb1679a" Jul 26 16:41:39.174933 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:39.174905 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:41:39.174989 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:39.174976 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:41:39.175026 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:39.175009 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:41:40.186564 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:40.186518 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" event={"ID":"e5f061ec-6ce6-4e9d-a0c2-ec564f607756","Type":"ContainerStarted","Data":"0e2d52f7179482517d90472a5e49850cc5f216533db430fd03b622357cdecc2e"} Jul 26 16:41:40.335065 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:40.335028 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:40.335294 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:40.335268 2563 patch_prober.go:28] interesting pod/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 26 16:41:40.335349 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:40.335334 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" podUID="e5f061ec-6ce6-4e9d-a0c2-ec564f607756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 26 16:41:41.335259 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:41.335205 2563 patch_prober.go:28] interesting pod/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 26 16:41:41.335655 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:41.335275 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" podUID="e5f061ec-6ce6-4e9d-a0c2-ec564f607756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 26 16:41:42.334382 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:42.334347 2563 patch_prober.go:28] interesting pod/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 26 16:41:42.334559 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:42.334395 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" podUID="e5f061ec-6ce6-4e9d-a0c2-ec564f607756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 26 16:41:43.335004 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:43.334961 2563 patch_prober.go:28] interesting pod/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 26 16:41:43.335442 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:43.335036 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" podUID="e5f061ec-6ce6-4e9d-a0c2-ec564f607756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 26 16:41:44.334889 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:44.334851 2563 patch_prober.go:28] interesting pod/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 26 16:41:44.335083 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:44.334907 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" podUID="e5f061ec-6ce6-4e9d-a0c2-ec564f607756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 26 16:41:45.338405 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:45.338372 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:45.338866 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:45.338647 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:41:45.339440 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:41:45.339424 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:42:23.478559 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:23.478520 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c"] Jul 26 16:42:23.479187 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:23.478776 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" podUID="ab9494e7-b55e-496c-8541-8554ce8fe83e" containerName="maas-api" containerID="cri-o://885292e9fc479bc9459f9041034ada28c6a1c0f8c41131af333ef16f5f46c9dc" gracePeriod=30 Jul 26 16:42:23.740458 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:23.740385 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" Jul 26 16:42:23.892348 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:23.892318 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab9494e7-b55e-496c-8541-8554ce8fe83e-maas-api-tls\") pod \"ab9494e7-b55e-496c-8541-8554ce8fe83e\" (UID: \"ab9494e7-b55e-496c-8541-8554ce8fe83e\") " Jul 26 16:42:23.892547 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:23.892429 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvzxx\" (UniqueName: \"kubernetes.io/projected/ab9494e7-b55e-496c-8541-8554ce8fe83e-kube-api-access-rvzxx\") pod \"ab9494e7-b55e-496c-8541-8554ce8fe83e\" (UID: \"ab9494e7-b55e-496c-8541-8554ce8fe83e\") " Jul 26 16:42:23.892547 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:23.892468 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab9494e7-b55e-496c-8541-8554ce8fe83e-openshift-service-ca\") pod \"ab9494e7-b55e-496c-8541-8554ce8fe83e\" (UID: \"ab9494e7-b55e-496c-8541-8554ce8fe83e\") " Jul 26 16:42:23.892961 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:23.892926 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab9494e7-b55e-496c-8541-8554ce8fe83e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab9494e7-b55e-496c-8541-8554ce8fe83e" (UID: "ab9494e7-b55e-496c-8541-8554ce8fe83e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:42:23.894590 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:23.894560 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab9494e7-b55e-496c-8541-8554ce8fe83e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab9494e7-b55e-496c-8541-8554ce8fe83e" (UID: "ab9494e7-b55e-496c-8541-8554ce8fe83e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:42:23.894781 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:23.894759 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9494e7-b55e-496c-8541-8554ce8fe83e-kube-api-access-rvzxx" (OuterVolumeSpecName: "kube-api-access-rvzxx") pod "ab9494e7-b55e-496c-8541-8554ce8fe83e" (UID: "ab9494e7-b55e-496c-8541-8554ce8fe83e"). InnerVolumeSpecName "kube-api-access-rvzxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:42:23.994007 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:23.993907 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab9494e7-b55e-496c-8541-8554ce8fe83e-maas-api-tls\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:42:23.994007 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:23.993941 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvzxx\" (UniqueName: \"kubernetes.io/projected/ab9494e7-b55e-496c-8541-8554ce8fe83e-kube-api-access-rvzxx\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:42:23.994007 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:23.993950 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab9494e7-b55e-496c-8541-8554ce8fe83e-openshift-service-ca\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:42:24.422153 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:24.422116 2563 generic.go:356] "Generic (PLEG): container finished" podID="ab9494e7-b55e-496c-8541-8554ce8fe83e" containerID="885292e9fc479bc9459f9041034ada28c6a1c0f8c41131af333ef16f5f46c9dc" exitCode=0 Jul 26 16:42:24.422378 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:24.422162 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" Jul 26 16:42:24.422378 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:24.422199 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" event={"ID":"ab9494e7-b55e-496c-8541-8554ce8fe83e","Type":"ContainerDied","Data":"885292e9fc479bc9459f9041034ada28c6a1c0f8c41131af333ef16f5f46c9dc"} Jul 26 16:42:24.422378 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:24.422273 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c" event={"ID":"ab9494e7-b55e-496c-8541-8554ce8fe83e","Type":"ContainerDied","Data":"e08ac911cbb55ca5b1588c471a26e0c066bef2b7663bbbfa9fa635fb0aeb391b"} Jul 26 16:42:24.422378 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:24.422298 2563 scope.go:117] "RemoveContainer" containerID="885292e9fc479bc9459f9041034ada28c6a1c0f8c41131af333ef16f5f46c9dc" Jul 26 16:42:24.433882 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:24.433861 2563 scope.go:117] "RemoveContainer" containerID="885292e9fc479bc9459f9041034ada28c6a1c0f8c41131af333ef16f5f46c9dc" Jul 26 16:42:24.434193 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:42:24.434172 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"885292e9fc479bc9459f9041034ada28c6a1c0f8c41131af333ef16f5f46c9dc\": container with ID starting with 885292e9fc479bc9459f9041034ada28c6a1c0f8c41131af333ef16f5f46c9dc not found: ID does not exist" containerID="885292e9fc479bc9459f9041034ada28c6a1c0f8c41131af333ef16f5f46c9dc" Jul 26 16:42:24.434276 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:24.434202 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"885292e9fc479bc9459f9041034ada28c6a1c0f8c41131af333ef16f5f46c9dc"} err="failed to get container status \"885292e9fc479bc9459f9041034ada28c6a1c0f8c41131af333ef16f5f46c9dc\": rpc error: code = NotFound desc = could not find container \"885292e9fc479bc9459f9041034ada28c6a1c0f8c41131af333ef16f5f46c9dc\": container with ID starting with 885292e9fc479bc9459f9041034ada28c6a1c0f8c41131af333ef16f5f46c9dc not found: ID does not exist" Jul 26 16:42:24.447427 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:24.447395 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c"] Jul 26 16:42:24.453770 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:24.453747 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ffa73817-76b7d96646-xqw9c"] Jul 26 16:42:24.891617 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:42:24.891582 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab9494e7-b55e-496c-8541-8554ce8fe83e" path="/var/lib/kubelet/pods/ab9494e7-b55e-496c-8541-8554ce8fe83e/volumes" Jul 26 16:43:34.615202 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:34.615120 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q"] Jul 26 16:43:34.615748 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:34.615354 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" podUID="e5f061ec-6ce6-4e9d-a0c2-ec564f607756" containerName="istio-proxy" containerID="cri-o://0e2d52f7179482517d90472a5e49850cc5f216533db430fd03b622357cdecc2e" gracePeriod=30 Jul 26 16:43:39.823564 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.823517 2563 generic.go:356] "Generic (PLEG): container finished" podID="e5f061ec-6ce6-4e9d-a0c2-ec564f607756" containerID="0e2d52f7179482517d90472a5e49850cc5f216533db430fd03b622357cdecc2e" exitCode=0 Jul 26 16:43:39.824035 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.823586 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" event={"ID":"e5f061ec-6ce6-4e9d-a0c2-ec564f607756","Type":"ContainerDied","Data":"0e2d52f7179482517d90472a5e49850cc5f216533db430fd03b622357cdecc2e"} Jul 26 16:43:39.824035 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.823650 2563 scope.go:117] "RemoveContainer" containerID="95fa8497ac4d87d7e56d98adcf782ee19dd893f9d12a5f8ce0301d229cb1679a" Jul 26 16:43:39.875588 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.875564 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:43:39.960231 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.960134 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-data\") pod \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " Jul 26 16:43:39.960231 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.960207 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-envoy\") pod \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " Jul 26 16:43:39.960483 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.960251 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-credential-socket\") pod \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " Jul 26 16:43:39.960483 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.960312 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istiod-ca-cert\") pod \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " Jul 26 16:43:39.960483 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.960361 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-workload-certs\") pod \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " Jul 26 16:43:39.960483 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.960384 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmrb8\" (UniqueName: \"kubernetes.io/projected/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-kube-api-access-jmrb8\") pod \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " Jul 26 16:43:39.960483 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.960424 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-workload-socket\") pod \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " Jul 26 16:43:39.960756 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.960679 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5f061ec-6ce6-4e9d-a0c2-ec564f607756" (UID: "e5f061ec-6ce6-4e9d-a0c2-ec564f607756"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:43:39.960756 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.960692 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5f061ec-6ce6-4e9d-a0c2-ec564f607756" (UID: "e5f061ec-6ce6-4e9d-a0c2-ec564f607756"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:43:39.960756 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.960702 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5f061ec-6ce6-4e9d-a0c2-ec564f607756" (UID: "e5f061ec-6ce6-4e9d-a0c2-ec564f607756"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:43:39.960756 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.960735 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-podinfo\") pod \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " Jul 26 16:43:39.960968 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.960808 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-token\") pod \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\" (UID: \"e5f061ec-6ce6-4e9d-a0c2-ec564f607756\") " Jul 26 16:43:39.960968 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.960950 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5f061ec-6ce6-4e9d-a0c2-ec564f607756" (UID: "e5f061ec-6ce6-4e9d-a0c2-ec564f607756"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:43:39.961179 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.961128 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istiod-ca-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:43:39.961179 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.961153 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-workload-certs\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:43:39.961179 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.961158 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5f061ec-6ce6-4e9d-a0c2-ec564f607756" (UID: "e5f061ec-6ce6-4e9d-a0c2-ec564f607756"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:43:39.961434 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.961167 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-workload-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:43:39.961434 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.961204 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-credential-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:43:39.962916 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.962882 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5f061ec-6ce6-4e9d-a0c2-ec564f607756" (UID: "e5f061ec-6ce6-4e9d-a0c2-ec564f607756"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:43:39.963152 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.963119 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5f061ec-6ce6-4e9d-a0c2-ec564f607756" (UID: "e5f061ec-6ce6-4e9d-a0c2-ec564f607756"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:43:39.963287 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.963194 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-kube-api-access-jmrb8" (OuterVolumeSpecName: "kube-api-access-jmrb8") pod "e5f061ec-6ce6-4e9d-a0c2-ec564f607756" (UID: "e5f061ec-6ce6-4e9d-a0c2-ec564f607756"). InnerVolumeSpecName "kube-api-access-jmrb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:43:39.963287 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:39.963274 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5f061ec-6ce6-4e9d-a0c2-ec564f607756" (UID: "e5f061ec-6ce6-4e9d-a0c2-ec564f607756"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:43:40.061938 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:40.061900 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-envoy\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:43:40.061938 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:40.061928 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmrb8\" (UniqueName: \"kubernetes.io/projected/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-kube-api-access-jmrb8\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:43:40.061938 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:40.061937 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-podinfo\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:43:40.061938 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:40.061945 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-token\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:43:40.062388 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:40.061953 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f061ec-6ce6-4e9d-a0c2-ec564f607756-istio-data\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:43:40.830345 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:40.830316 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" Jul 26 16:43:40.830748 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:40.830303 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q" event={"ID":"e5f061ec-6ce6-4e9d-a0c2-ec564f607756","Type":"ContainerDied","Data":"29bcec8867592b7e8f9d2d0fd418a133c808e98b5d82bf63d395ca18a4b8dc37"} Jul 26 16:43:40.830748 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:40.830459 2563 scope.go:117] "RemoveContainer" containerID="0e2d52f7179482517d90472a5e49850cc5f216533db430fd03b622357cdecc2e" Jul 26 16:43:40.865888 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:40.865854 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q"] Jul 26 16:43:40.873333 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:40.873302 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ffa73817-openshift-default-6ddb6b7fdd-xkr6q"] Jul 26 16:43:40.892150 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:40.892123 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5f061ec-6ce6-4e9d-a0c2-ec564f607756" path="/var/lib/kubelet/pods/e5f061ec-6ce6-4e9d-a0c2-ec564f607756/volumes" Jul 26 16:43:47.083453 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:43:47.083416 2563 scope.go:117] "RemoveContainer" containerID="952f482f85942cd472540ce6f50f67a4e4db2bb433e28244319bf677151ff0d0" Jul 26 16:44:34.491117 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.491078 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788"] Jul 26 16:44:34.491909 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.491462 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5f061ec-6ce6-4e9d-a0c2-ec564f607756" containerName="istio-proxy" Jul 26 16:44:34.491909 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.491474 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5f061ec-6ce6-4e9d-a0c2-ec564f607756" containerName="istio-proxy" Jul 26 16:44:34.491909 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.491484 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5f061ec-6ce6-4e9d-a0c2-ec564f607756" containerName="istio-proxy" Jul 26 16:44:34.491909 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.491490 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5f061ec-6ce6-4e9d-a0c2-ec564f607756" containerName="istio-proxy" Jul 26 16:44:34.491909 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.491499 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab9494e7-b55e-496c-8541-8554ce8fe83e" containerName="maas-api" Jul 26 16:44:34.491909 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.491504 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab9494e7-b55e-496c-8541-8554ce8fe83e" containerName="maas-api" Jul 26 16:44:34.491909 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.491573 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab9494e7-b55e-496c-8541-8554ce8fe83e" containerName="maas-api" Jul 26 16:44:34.491909 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.491583 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5f061ec-6ce6-4e9d-a0c2-ec564f607756" containerName="istio-proxy" Jul 26 16:44:34.491909 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.491589 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5f061ec-6ce6-4e9d-a0c2-ec564f607756" containerName="istio-proxy" Jul 26 16:44:34.494739 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.494719 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.498326 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.498302 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e38070-openshift-default-dockercfg-55h7z\"" Jul 26 16:44:34.508853 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.508828 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788"] Jul 26 16:44:34.538488 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.538449 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119c46c3-de88-417e-86a2-e8c808b1e339-istiod-ca-cert\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.538662 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.538497 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-workload-socket\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.538662 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.538546 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-istio-data\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.538662 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.538571 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-credential-socket\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.538662 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.538589 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-istio-envoy\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.538662 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.538610 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119c46c3-de88-417e-86a2-e8c808b1e339-istio-podinfo\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.538901 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.538696 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119c46c3-de88-417e-86a2-e8c808b1e339-istio-token\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.538901 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.538742 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-workload-certs\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.538901 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.538788 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxnbs\" (UniqueName: \"kubernetes.io/projected/119c46c3-de88-417e-86a2-e8c808b1e339-kube-api-access-gxnbs\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.639308 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.639254 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119c46c3-de88-417e-86a2-e8c808b1e339-istiod-ca-cert\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.639308 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.639316 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-workload-socket\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.639582 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.639359 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-istio-data\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.639582 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.639383 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-credential-socket\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.639582 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.639408 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-istio-envoy\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.639582 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.639436 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119c46c3-de88-417e-86a2-e8c808b1e339-istio-podinfo\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.639582 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.639474 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119c46c3-de88-417e-86a2-e8c808b1e339-istio-token\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.639582 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.639507 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-workload-certs\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.639582 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.639554 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxnbs\" (UniqueName: \"kubernetes.io/projected/119c46c3-de88-417e-86a2-e8c808b1e339-kube-api-access-gxnbs\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.639962 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.639804 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-credential-socket\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.639962 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.639889 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-workload-socket\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.640076 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.639997 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-workload-certs\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.640142 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.640120 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119c46c3-de88-417e-86a2-e8c808b1e339-istiod-ca-cert\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.640202 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.640186 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-istio-data\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.641794 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.641772 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-istio-envoy\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.641939 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.641921 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119c46c3-de88-417e-86a2-e8c808b1e339-istio-podinfo\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.651092 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.651067 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119c46c3-de88-417e-86a2-e8c808b1e339-istio-token\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.651195 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.651185 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxnbs\" (UniqueName: \"kubernetes.io/projected/119c46c3-de88-417e-86a2-e8c808b1e339-kube-api-access-gxnbs\") pod \"e2e-inf-a-e38070-openshift-default-545f994bc6-hm788\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.806015 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.805960 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:34.960095 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.960065 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788"] Jul 26 16:44:34.961699 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:44:34.961675 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod119c46c3_de88_417e_86a2_e8c808b1e339.slice/crio-4f2d5c12092e1b04f918bcd12b08dff2786f79497b7de9e946f7103c25ab3989 WatchSource:0}: Error finding container 4f2d5c12092e1b04f918bcd12b08dff2786f79497b7de9e946f7103c25ab3989: Status 404 returned error can't find the container with id 4f2d5c12092e1b04f918bcd12b08dff2786f79497b7de9e946f7103c25ab3989 Jul 26 16:44:34.964399 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.964363 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:44:34.964468 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.964433 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:44:34.964516 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:34.964469 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:44:35.113940 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:35.113851 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" event={"ID":"119c46c3-de88-417e-86a2-e8c808b1e339","Type":"ContainerStarted","Data":"872f7e9d6b6f390455fb43b7634c1679395cf77452dd9ad4dd727cf567b826c3"} Jul 26 16:44:35.113940 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:35.113923 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" event={"ID":"119c46c3-de88-417e-86a2-e8c808b1e339","Type":"ContainerStarted","Data":"4f2d5c12092e1b04f918bcd12b08dff2786f79497b7de9e946f7103c25ab3989"} Jul 26 16:44:35.135694 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:35.135639 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" podStartSLOduration=1.135622919 podStartE2EDuration="1.135622919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:44:35.135046971 +0000 UTC m=+2148.716133544" watchObservedRunningTime="2026-07-26 16:44:35.135622919 +0000 UTC m=+2148.716709489" Jul 26 16:44:35.806669 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:35.806630 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:36.811327 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:36.811293 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:37.125327 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:37.125189 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:37.126261 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:37.126244 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:37.171394 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:37.171359 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd"] Jul 26 16:44:37.174830 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:37.174806 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" Jul 26 16:44:37.178154 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:37.177936 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e38070-serving-cert\"" Jul 26 16:44:37.186668 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:37.186643 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd"] Jul 26 16:44:37.269722 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:37.269687 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a1a527-6b02-44f0-be04-dd1ac98a60b2-maas-api-tls\") pod \"maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd\" (UID: \"41a1a527-6b02-44f0-be04-dd1ac98a60b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" Jul 26 16:44:37.269897 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:37.269739 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a1a527-6b02-44f0-be04-dd1ac98a60b2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd\" (UID: \"41a1a527-6b02-44f0-be04-dd1ac98a60b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" Jul 26 16:44:37.269897 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:37.269845 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjvsx\" (UniqueName: \"kubernetes.io/projected/41a1a527-6b02-44f0-be04-dd1ac98a60b2-kube-api-access-tjvsx\") pod \"maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd\" (UID: \"41a1a527-6b02-44f0-be04-dd1ac98a60b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" Jul 26 16:44:37.371070 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:37.371032 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a1a527-6b02-44f0-be04-dd1ac98a60b2-maas-api-tls\") pod \"maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd\" (UID: \"41a1a527-6b02-44f0-be04-dd1ac98a60b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" Jul 26 16:44:37.371259 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:37.371078 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a1a527-6b02-44f0-be04-dd1ac98a60b2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd\" (UID: \"41a1a527-6b02-44f0-be04-dd1ac98a60b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" Jul 26 16:44:37.371259 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:37.371106 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjvsx\" (UniqueName: \"kubernetes.io/projected/41a1a527-6b02-44f0-be04-dd1ac98a60b2-kube-api-access-tjvsx\") pod \"maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd\" (UID: \"41a1a527-6b02-44f0-be04-dd1ac98a60b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" Jul 26 16:44:37.371259 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:44:37.371191 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-serving-cert: secret "maas-api-e2e-inf-a-e38070-serving-cert" not found Jul 26 16:44:37.371374 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:44:37.371283 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a1a527-6b02-44f0-be04-dd1ac98a60b2-maas-api-tls podName:41a1a527-6b02-44f0-be04-dd1ac98a60b2 nodeName:}" failed. No retries permitted until 2026-07-26 16:44:37.871265361 +0000 UTC m=+2151.452351909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41a1a527-6b02-44f0-be04-dd1ac98a60b2-maas-api-tls") pod "maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" (UID: "41a1a527-6b02-44f0-be04-dd1ac98a60b2") : secret "maas-api-e2e-inf-a-e38070-serving-cert" not found Jul 26 16:44:37.371749 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:37.371725 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a1a527-6b02-44f0-be04-dd1ac98a60b2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd\" (UID: \"41a1a527-6b02-44f0-be04-dd1ac98a60b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" Jul 26 16:44:37.380661 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:37.380596 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjvsx\" (UniqueName: \"kubernetes.io/projected/41a1a527-6b02-44f0-be04-dd1ac98a60b2-kube-api-access-tjvsx\") pod \"maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd\" (UID: \"41a1a527-6b02-44f0-be04-dd1ac98a60b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" Jul 26 16:44:37.876315 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:37.876279 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a1a527-6b02-44f0-be04-dd1ac98a60b2-maas-api-tls\") pod \"maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd\" (UID: \"41a1a527-6b02-44f0-be04-dd1ac98a60b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" Jul 26 16:44:37.878802 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:37.878775 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a1a527-6b02-44f0-be04-dd1ac98a60b2-maas-api-tls\") pod \"maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd\" (UID: \"41a1a527-6b02-44f0-be04-dd1ac98a60b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" Jul 26 16:44:38.090623 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:38.090587 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" Jul 26 16:44:38.246567 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:38.246537 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd"] Jul 26 16:44:38.248199 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:44:38.248171 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a1a527_6b02_44f0_be04_dd1ac98a60b2.slice/crio-2ba3d2a9c8d1e41e3e1c0ffa196ff7ee8449c16ae27430043112b73ab1bac092 WatchSource:0}: Error finding container 2ba3d2a9c8d1e41e3e1c0ffa196ff7ee8449c16ae27430043112b73ab1bac092: Status 404 returned error can't find the container with id 2ba3d2a9c8d1e41e3e1c0ffa196ff7ee8449c16ae27430043112b73ab1bac092 Jul 26 16:44:39.138297 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:39.138263 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" event={"ID":"41a1a527-6b02-44f0-be04-dd1ac98a60b2","Type":"ContainerStarted","Data":"09fa119fc2fa577b2c69350ed0e4c6b0eda3c7dcbd807689a5b291bdb89126df"} Jul 26 16:44:39.138690 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:39.138319 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" event={"ID":"41a1a527-6b02-44f0-be04-dd1ac98a60b2","Type":"ContainerStarted","Data":"2ba3d2a9c8d1e41e3e1c0ffa196ff7ee8449c16ae27430043112b73ab1bac092"} Jul 26 16:44:39.138690 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:39.138499 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" Jul 26 16:44:39.159804 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:39.159729 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" podStartSLOduration=1.6771619260000001 podStartE2EDuration="2.159713291s" totalImagesPullingTime="482.551365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:44:39.158659152 +0000 UTC m=+2152.739745722" watchObservedRunningTime="2026-07-26 16:44:39.159713291 +0000 UTC m=+2152.740799861" Jul 26 16:44:42.278247 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.269049 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx"] Jul 26 16:44:42.286252 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.284734 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.288614 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.288538 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6a9b62-openshift-default-dockercfg-hv4ng\"" Jul 26 16:44:42.289203 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.289186 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx"] Jul 26 16:44:42.315937 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.315907 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc6a646-d246-4596-a570-089ed4b2771a-istio-token\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.316111 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.315953 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-workload-certs\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.316111 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.316027 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-istio-envoy\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.316111 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.316058 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-istio-data\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.316111 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.316094 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srm7s\" (UniqueName: \"kubernetes.io/projected/2dc6a646-d246-4596-a570-089ed4b2771a-kube-api-access-srm7s\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.316363 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.316159 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-workload-socket\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.316363 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.316262 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc6a646-d246-4596-a570-089ed4b2771a-istiod-ca-cert\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.316363 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.316305 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-credential-socket\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.316363 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.316338 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc6a646-d246-4596-a570-089ed4b2771a-istio-podinfo\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.417010 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.416967 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc6a646-d246-4596-a570-089ed4b2771a-istio-podinfo\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.417198 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.417023 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc6a646-d246-4596-a570-089ed4b2771a-istio-token\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.417198 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.417047 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-workload-certs\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.417198 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.417087 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-istio-envoy\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.417198 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.417107 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-istio-data\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.417198 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.417132 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srm7s\" (UniqueName: \"kubernetes.io/projected/2dc6a646-d246-4596-a570-089ed4b2771a-kube-api-access-srm7s\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.417198 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.417152 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-workload-socket\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.417198 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.417173 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc6a646-d246-4596-a570-089ed4b2771a-istiod-ca-cert\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.417439 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.417204 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-credential-socket\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.417736 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.417707 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-credential-socket\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.417861 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.417820 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-workload-socket\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.417927 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.417863 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-workload-certs\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.418024 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.418005 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-istio-data\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.418309 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.418284 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc6a646-d246-4596-a570-089ed4b2771a-istiod-ca-cert\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.419661 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.419641 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-istio-envoy\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.420673 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.420621 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc6a646-d246-4596-a570-089ed4b2771a-istio-podinfo\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.436024 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.435997 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc6a646-d246-4596-a570-089ed4b2771a-istio-token\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.436122 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.436108 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srm7s\" (UniqueName: \"kubernetes.io/projected/2dc6a646-d246-4596-a570-089ed4b2771a-kube-api-access-srm7s\") pod \"e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.598023 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.597931 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:42.784347 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.783695 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx"] Jul 26 16:44:42.785295 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:44:42.785256 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc6a646_d246_4596_a570_089ed4b2771a.slice/crio-ffe787b16557cfeb0acd0ab7fa2ea97eb32e4943055aeb09b5da0965fbc7ecf6 WatchSource:0}: Error finding container ffe787b16557cfeb0acd0ab7fa2ea97eb32e4943055aeb09b5da0965fbc7ecf6: Status 404 returned error can't find the container with id ffe787b16557cfeb0acd0ab7fa2ea97eb32e4943055aeb09b5da0965fbc7ecf6 Jul 26 16:44:42.788456 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.788423 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:44:42.788559 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.788500 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:44:42.788559 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:42.788543 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:44:43.163698 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:43.163651 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" event={"ID":"2dc6a646-d246-4596-a570-089ed4b2771a","Type":"ContainerStarted","Data":"dff37e89c3d474607fa3f75fbac2a9f5a8102e4cfad0d61e38a658ba1306fdd3"} Jul 26 16:44:43.163914 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:43.163727 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" event={"ID":"2dc6a646-d246-4596-a570-089ed4b2771a","Type":"ContainerStarted","Data":"ffe787b16557cfeb0acd0ab7fa2ea97eb32e4943055aeb09b5da0965fbc7ecf6"} Jul 26 16:44:43.189715 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:43.189631 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" podStartSLOduration=1.189608382 podStartE2EDuration="1.189608382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:44:43.188505147 +0000 UTC m=+2156.769591741" watchObservedRunningTime="2026-07-26 16:44:43.189608382 +0000 UTC m=+2156.770694952" Jul 26 16:44:43.599135 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:43.599091 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:43.604612 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:43.604583 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:44.172948 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:44.172907 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:44.177738 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:44.177705 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:45.148791 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:45.148764 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" Jul 26 16:44:47.107136 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:47.107106 2563 scope.go:117] "RemoveContainer" containerID="35b9fe21e958145aadf59aeba76af7449969cea4bbfea255fdb7233651556b48" Jul 26 16:44:49.205796 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:49.205762 2563 generic.go:356] "Generic (PLEG): container finished" podID="119c46c3-de88-417e-86a2-e8c808b1e339" containerID="872f7e9d6b6f390455fb43b7634c1679395cf77452dd9ad4dd727cf567b826c3" exitCode=0 Jul 26 16:44:49.206252 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:49.205837 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" event={"ID":"119c46c3-de88-417e-86a2-e8c808b1e339","Type":"ContainerDied","Data":"872f7e9d6b6f390455fb43b7634c1679395cf77452dd9ad4dd727cf567b826c3"} Jul 26 16:44:49.206405 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:49.206387 2563 scope.go:117] "RemoveContainer" containerID="872f7e9d6b6f390455fb43b7634c1679395cf77452dd9ad4dd727cf567b826c3" Jul 26 16:44:49.207444 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:49.207415 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:44:49.207514 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:49.207482 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:44:49.207555 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:49.207514 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:44:50.215830 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:50.215779 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" event={"ID":"119c46c3-de88-417e-86a2-e8c808b1e339","Type":"ContainerStarted","Data":"22b75812ebfb580867685902bbcbae7ce856ecf9744501cdc301652f23344363"} Jul 26 16:44:50.806340 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:50.806297 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:50.807391 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:50.807356 2563 patch_prober.go:28] interesting pod/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 26 16:44:50.807548 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:50.807427 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" podUID="119c46c3-de88-417e-86a2-e8c808b1e339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 26 16:44:51.806773 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:51.806732 2563 patch_prober.go:28] interesting pod/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 26 16:44:51.807409 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:51.807348 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" podUID="119c46c3-de88-417e-86a2-e8c808b1e339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 26 16:44:52.125386 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:52.125287 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:52.806945 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:52.806897 2563 patch_prober.go:28] interesting pod/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 26 16:44:52.807480 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:52.806967 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" podUID="119c46c3-de88-417e-86a2-e8c808b1e339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 26 16:44:53.806606 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:53.806564 2563 patch_prober.go:28] interesting pod/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 26 16:44:53.806797 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:53.806635 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" podUID="119c46c3-de88-417e-86a2-e8c808b1e339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 26 16:44:54.810924 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:54.810893 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:54.812153 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:54.812127 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:44:57.258363 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:57.258324 2563 generic.go:356] "Generic (PLEG): container finished" podID="2dc6a646-d246-4596-a570-089ed4b2771a" containerID="dff37e89c3d474607fa3f75fbac2a9f5a8102e4cfad0d61e38a658ba1306fdd3" exitCode=0 Jul 26 16:44:57.258947 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:57.258379 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" event={"ID":"2dc6a646-d246-4596-a570-089ed4b2771a","Type":"ContainerDied","Data":"dff37e89c3d474607fa3f75fbac2a9f5a8102e4cfad0d61e38a658ba1306fdd3"} Jul 26 16:44:57.258947 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:57.258918 2563 scope.go:117] "RemoveContainer" containerID="dff37e89c3d474607fa3f75fbac2a9f5a8102e4cfad0d61e38a658ba1306fdd3" Jul 26 16:44:57.260008 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:57.259980 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:44:57.260074 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:57.260049 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:44:57.260112 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:57.260086 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:44:58.269693 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:58.269651 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" event={"ID":"2dc6a646-d246-4596-a570-089ed4b2771a","Type":"ContainerStarted","Data":"12ae5636373a792b7cc13b19952f5847f81f3870d4c640ee74520290532a8349"} Jul 26 16:44:58.598622 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:58.598581 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:58.599858 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:58.599827 2563 patch_prober.go:28] interesting pod/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 26 16:44:58.599985 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:58.599896 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" podUID="2dc6a646-d246-4596-a570-089ed4b2771a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 26 16:44:59.173322 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:59.173285 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:44:59.598361 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:59.598323 2563 patch_prober.go:28] interesting pod/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 26 16:44:59.598780 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:44:59.598380 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" podUID="2dc6a646-d246-4596-a570-089ed4b2771a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 26 16:45:00.599209 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:45:00.599165 2563 patch_prober.go:28] interesting pod/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 26 16:45:00.599617 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:45:00.599270 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" podUID="2dc6a646-d246-4596-a570-089ed4b2771a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 26 16:45:01.598894 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:45:01.598856 2563 patch_prober.go:28] interesting pod/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 26 16:45:01.599089 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:45:01.598916 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" podUID="2dc6a646-d246-4596-a570-089ed4b2771a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 26 16:45:02.602709 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:45:02.602671 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:45:02.603703 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:45:02.603683 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:46:20.841356 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:20.841258 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e38070-txlzd"] Jul 26 16:46:20.844868 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:20.844847 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e38070-txlzd" Jul 26 16:46:20.847550 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:20.847526 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2xlcn\"" Jul 26 16:46:20.853188 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:20.853163 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e38070-txlzd"] Jul 26 16:46:20.934290 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:20.934253 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5668e27-2f2d-4416-9aff-03a2dece78f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e38070-txlzd\" (UID: \"f5668e27-2f2d-4416-9aff-03a2dece78f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e38070-txlzd" Jul 26 16:46:21.035286 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:21.035214 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5668e27-2f2d-4416-9aff-03a2dece78f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e38070-txlzd\" (UID: \"f5668e27-2f2d-4416-9aff-03a2dece78f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e38070-txlzd" Jul 26 16:46:21.035871 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:21.035848 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5668e27-2f2d-4416-9aff-03a2dece78f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e38070-txlzd\" (UID: \"f5668e27-2f2d-4416-9aff-03a2dece78f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e38070-txlzd" Jul 26 16:46:21.157994 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:21.157902 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e38070-txlzd" Jul 26 16:46:21.294159 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:21.294114 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e38070-txlzd"] Jul 26 16:46:21.295242 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:46:21.295200 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5668e27_2f2d_4416_9aff_03a2dece78f2.slice/crio-fd855dbc4009e3add864802b985176ab9bdda31d9330319e5d1ffd1716cc3fb7 WatchSource:0}: Error finding container fd855dbc4009e3add864802b985176ab9bdda31d9330319e5d1ffd1716cc3fb7: Status 404 returned error can't find the container with id fd855dbc4009e3add864802b985176ab9bdda31d9330319e5d1ffd1716cc3fb7 Jul 26 16:46:21.726916 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:21.726819 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e38070-txlzd" event={"ID":"f5668e27-2f2d-4416-9aff-03a2dece78f2","Type":"ContainerStarted","Data":"14cb1d3c627495c04ab633bef045ed732e95d428a15a45fce195a5773c61e95c"} Jul 26 16:46:21.726916 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:21.726883 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e38070-txlzd" event={"ID":"f5668e27-2f2d-4416-9aff-03a2dece78f2","Type":"ContainerStarted","Data":"fd855dbc4009e3add864802b985176ab9bdda31d9330319e5d1ffd1716cc3fb7"} Jul 26 16:46:21.746265 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:21.746193 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e38070-txlzd" podStartSLOduration=1.746178559 podStartE2EDuration="1.746178559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:46:21.742980462 +0000 UTC m=+2255.324067047" watchObservedRunningTime="2026-07-26 16:46:21.746178559 +0000 UTC m=+2255.327265129" Jul 26 16:46:22.733654 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:22.733616 2563 generic.go:356] "Generic (PLEG): container finished" podID="f5668e27-2f2d-4416-9aff-03a2dece78f2" containerID="14cb1d3c627495c04ab633bef045ed732e95d428a15a45fce195a5773c61e95c" exitCode=0 Jul 26 16:46:22.734052 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:22.733669 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e38070-txlzd" event={"ID":"f5668e27-2f2d-4416-9aff-03a2dece78f2","Type":"ContainerDied","Data":"14cb1d3c627495c04ab633bef045ed732e95d428a15a45fce195a5773c61e95c"} Jul 26 16:46:23.875773 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:23.875748 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e38070-txlzd" Jul 26 16:46:23.961623 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:23.961585 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5668e27-2f2d-4416-9aff-03a2dece78f2-maas-api-service-ca\") pod \"f5668e27-2f2d-4416-9aff-03a2dece78f2\" (UID: \"f5668e27-2f2d-4416-9aff-03a2dece78f2\") " Jul 26 16:46:23.961990 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:23.961965 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5668e27-2f2d-4416-9aff-03a2dece78f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5668e27-2f2d-4416-9aff-03a2dece78f2" (UID: "f5668e27-2f2d-4416-9aff-03a2dece78f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:46:24.063200 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:24.063155 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5668e27-2f2d-4416-9aff-03a2dece78f2-maas-api-service-ca\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:46:24.746829 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:24.746798 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e38070-txlzd" Jul 26 16:46:24.746829 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:24.746821 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e38070-txlzd" event={"ID":"f5668e27-2f2d-4416-9aff-03a2dece78f2","Type":"ContainerDied","Data":"fd855dbc4009e3add864802b985176ab9bdda31d9330319e5d1ffd1716cc3fb7"} Jul 26 16:46:24.747040 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:24.746873 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd855dbc4009e3add864802b985176ab9bdda31d9330319e5d1ffd1716cc3fb7" Jul 26 16:46:31.330625 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:31.330592 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd"] Jul 26 16:46:31.331188 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:31.330841 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" podUID="41a1a527-6b02-44f0-be04-dd1ac98a60b2" containerName="maas-api" containerID="cri-o://09fa119fc2fa577b2c69350ed0e4c6b0eda3c7dcbd807689a5b291bdb89126df" gracePeriod=30 Jul 26 16:46:31.786895 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:31.786862 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" Jul 26 16:46:31.787694 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:31.787663 2563 generic.go:356] "Generic (PLEG): container finished" podID="41a1a527-6b02-44f0-be04-dd1ac98a60b2" containerID="09fa119fc2fa577b2c69350ed0e4c6b0eda3c7dcbd807689a5b291bdb89126df" exitCode=0 Jul 26 16:46:31.787805 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:31.787712 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" event={"ID":"41a1a527-6b02-44f0-be04-dd1ac98a60b2","Type":"ContainerDied","Data":"09fa119fc2fa577b2c69350ed0e4c6b0eda3c7dcbd807689a5b291bdb89126df"} Jul 26 16:46:31.787805 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:31.787773 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" event={"ID":"41a1a527-6b02-44f0-be04-dd1ac98a60b2","Type":"ContainerDied","Data":"2ba3d2a9c8d1e41e3e1c0ffa196ff7ee8449c16ae27430043112b73ab1bac092"} Jul 26 16:46:31.787896 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:31.787809 2563 scope.go:117] "RemoveContainer" containerID="09fa119fc2fa577b2c69350ed0e4c6b0eda3c7dcbd807689a5b291bdb89126df" Jul 26 16:46:31.799972 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:31.799956 2563 scope.go:117] "RemoveContainer" containerID="09fa119fc2fa577b2c69350ed0e4c6b0eda3c7dcbd807689a5b291bdb89126df" Jul 26 16:46:31.800315 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:46:31.800294 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09fa119fc2fa577b2c69350ed0e4c6b0eda3c7dcbd807689a5b291bdb89126df\": container with ID starting with 09fa119fc2fa577b2c69350ed0e4c6b0eda3c7dcbd807689a5b291bdb89126df not found: ID does not exist" containerID="09fa119fc2fa577b2c69350ed0e4c6b0eda3c7dcbd807689a5b291bdb89126df" Jul 26 16:46:31.800396 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:31.800324 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09fa119fc2fa577b2c69350ed0e4c6b0eda3c7dcbd807689a5b291bdb89126df"} err="failed to get container status \"09fa119fc2fa577b2c69350ed0e4c6b0eda3c7dcbd807689a5b291bdb89126df\": rpc error: code = NotFound desc = could not find container \"09fa119fc2fa577b2c69350ed0e4c6b0eda3c7dcbd807689a5b291bdb89126df\": container with ID starting with 09fa119fc2fa577b2c69350ed0e4c6b0eda3c7dcbd807689a5b291bdb89126df not found: ID does not exist" Jul 26 16:46:31.940040 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:31.939943 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjvsx\" (UniqueName: \"kubernetes.io/projected/41a1a527-6b02-44f0-be04-dd1ac98a60b2-kube-api-access-tjvsx\") pod \"41a1a527-6b02-44f0-be04-dd1ac98a60b2\" (UID: \"41a1a527-6b02-44f0-be04-dd1ac98a60b2\") " Jul 26 16:46:31.940040 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:31.939983 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a1a527-6b02-44f0-be04-dd1ac98a60b2-openshift-service-ca\") pod \"41a1a527-6b02-44f0-be04-dd1ac98a60b2\" (UID: \"41a1a527-6b02-44f0-be04-dd1ac98a60b2\") " Jul 26 16:46:31.940314 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:31.940078 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a1a527-6b02-44f0-be04-dd1ac98a60b2-maas-api-tls\") pod \"41a1a527-6b02-44f0-be04-dd1ac98a60b2\" (UID: \"41a1a527-6b02-44f0-be04-dd1ac98a60b2\") " Jul 26 16:46:31.940466 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:31.940444 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a1a527-6b02-44f0-be04-dd1ac98a60b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "41a1a527-6b02-44f0-be04-dd1ac98a60b2" (UID: "41a1a527-6b02-44f0-be04-dd1ac98a60b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:46:31.942172 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:31.942146 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41a1a527-6b02-44f0-be04-dd1ac98a60b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "41a1a527-6b02-44f0-be04-dd1ac98a60b2" (UID: "41a1a527-6b02-44f0-be04-dd1ac98a60b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:46:31.942282 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:31.942203 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a1a527-6b02-44f0-be04-dd1ac98a60b2-kube-api-access-tjvsx" (OuterVolumeSpecName: "kube-api-access-tjvsx") pod "41a1a527-6b02-44f0-be04-dd1ac98a60b2" (UID: "41a1a527-6b02-44f0-be04-dd1ac98a60b2"). InnerVolumeSpecName "kube-api-access-tjvsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:46:32.041549 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:32.041500 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a1a527-6b02-44f0-be04-dd1ac98a60b2-maas-api-tls\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:46:32.041549 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:32.041542 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjvsx\" (UniqueName: \"kubernetes.io/projected/41a1a527-6b02-44f0-be04-dd1ac98a60b2-kube-api-access-tjvsx\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:46:32.041549 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:32.041552 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a1a527-6b02-44f0-be04-dd1ac98a60b2-openshift-service-ca\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:46:32.793205 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:32.793174 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd" Jul 26 16:46:32.816504 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:32.816470 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd"] Jul 26 16:46:32.820695 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:32.820670 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e38070-6d8ccfb87b-s54sd"] Jul 26 16:46:32.890324 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:32.890293 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41a1a527-6b02-44f0-be04-dd1ac98a60b2" path="/var/lib/kubelet/pods/41a1a527-6b02-44f0-be04-dd1ac98a60b2/volumes" Jul 26 16:46:35.921802 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:35.921766 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e38070-txlzd"] Jul 26 16:46:35.926101 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:35.926071 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e38070-txlzd"] Jul 26 16:46:36.890304 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:36.890265 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5668e27-2f2d-4416-9aff-03a2dece78f2" path="/var/lib/kubelet/pods/f5668e27-2f2d-4416-9aff-03a2dece78f2/volumes" Jul 26 16:46:42.238909 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:42.238875 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788"] Jul 26 16:46:42.239388 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:42.239140 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" podUID="119c46c3-de88-417e-86a2-e8c808b1e339" containerName="istio-proxy" containerID="cri-o://22b75812ebfb580867685902bbcbae7ce856ecf9744501cdc301652f23344363" gracePeriod=30 Jul 26 16:46:47.134699 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.134660 2563 scope.go:117] "RemoveContainer" containerID="e6229ee9225b83181ea3e26d059426c85d435922180c8702aed3ec77800bf83f" Jul 26 16:46:47.494956 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.494931 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:46:47.686371 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.686279 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-istio-envoy\") pod \"119c46c3-de88-417e-86a2-e8c808b1e339\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " Jul 26 16:46:47.686371 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.686326 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-credential-socket\") pod \"119c46c3-de88-417e-86a2-e8c808b1e339\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " Jul 26 16:46:47.686371 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.686343 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-workload-certs\") pod \"119c46c3-de88-417e-86a2-e8c808b1e339\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " Jul 26 16:46:47.686371 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.686362 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119c46c3-de88-417e-86a2-e8c808b1e339-istio-token\") pod \"119c46c3-de88-417e-86a2-e8c808b1e339\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " Jul 26 16:46:47.686682 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.686405 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119c46c3-de88-417e-86a2-e8c808b1e339-istiod-ca-cert\") pod \"119c46c3-de88-417e-86a2-e8c808b1e339\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " Jul 26 16:46:47.686682 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.686429 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-workload-socket\") pod \"119c46c3-de88-417e-86a2-e8c808b1e339\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " Jul 26 16:46:47.686682 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.686486 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-istio-data\") pod \"119c46c3-de88-417e-86a2-e8c808b1e339\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " Jul 26 16:46:47.686682 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.686511 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119c46c3-de88-417e-86a2-e8c808b1e339-istio-podinfo\") pod \"119c46c3-de88-417e-86a2-e8c808b1e339\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " Jul 26 16:46:47.686682 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.686549 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxnbs\" (UniqueName: \"kubernetes.io/projected/119c46c3-de88-417e-86a2-e8c808b1e339-kube-api-access-gxnbs\") pod \"119c46c3-de88-417e-86a2-e8c808b1e339\" (UID: \"119c46c3-de88-417e-86a2-e8c808b1e339\") " Jul 26 16:46:47.686682 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.686638 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "119c46c3-de88-417e-86a2-e8c808b1e339" (UID: "119c46c3-de88-417e-86a2-e8c808b1e339"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:46:47.686891 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.686702 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "119c46c3-de88-417e-86a2-e8c808b1e339" (UID: "119c46c3-de88-417e-86a2-e8c808b1e339"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:46:47.686935 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.686890 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/119c46c3-de88-417e-86a2-e8c808b1e339-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "119c46c3-de88-417e-86a2-e8c808b1e339" (UID: "119c46c3-de88-417e-86a2-e8c808b1e339"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:46:47.686935 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.686904 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-credential-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:46:47.686935 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.686925 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-workload-certs\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:46:47.686935 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.686922 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "119c46c3-de88-417e-86a2-e8c808b1e339" (UID: "119c46c3-de88-417e-86a2-e8c808b1e339"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:46:47.687511 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.687484 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-istio-data" (OuterVolumeSpecName: "istio-data") pod "119c46c3-de88-417e-86a2-e8c808b1e339" (UID: "119c46c3-de88-417e-86a2-e8c808b1e339"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:46:47.688925 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.688894 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119c46c3-de88-417e-86a2-e8c808b1e339-kube-api-access-gxnbs" (OuterVolumeSpecName: "kube-api-access-gxnbs") pod "119c46c3-de88-417e-86a2-e8c808b1e339" (UID: "119c46c3-de88-417e-86a2-e8c808b1e339"). InnerVolumeSpecName "kube-api-access-gxnbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:46:47.688925 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.688913 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "119c46c3-de88-417e-86a2-e8c808b1e339" (UID: "119c46c3-de88-417e-86a2-e8c808b1e339"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:46:47.689091 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.688956 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119c46c3-de88-417e-86a2-e8c808b1e339-istio-token" (OuterVolumeSpecName: "istio-token") pod "119c46c3-de88-417e-86a2-e8c808b1e339" (UID: "119c46c3-de88-417e-86a2-e8c808b1e339"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:46:47.689091 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.688979 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/119c46c3-de88-417e-86a2-e8c808b1e339-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "119c46c3-de88-417e-86a2-e8c808b1e339" (UID: "119c46c3-de88-417e-86a2-e8c808b1e339"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:46:47.787396 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.787364 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-istio-envoy\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:46:47.787396 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.787391 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119c46c3-de88-417e-86a2-e8c808b1e339-istio-token\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:46:47.787396 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.787400 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119c46c3-de88-417e-86a2-e8c808b1e339-istiod-ca-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:46:47.787624 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.787408 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-workload-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:46:47.787624 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.787416 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119c46c3-de88-417e-86a2-e8c808b1e339-istio-data\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:46:47.787624 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.787424 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119c46c3-de88-417e-86a2-e8c808b1e339-istio-podinfo\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:46:47.787624 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.787432 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxnbs\" (UniqueName: \"kubernetes.io/projected/119c46c3-de88-417e-86a2-e8c808b1e339-kube-api-access-gxnbs\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:46:47.874395 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.874359 2563 generic.go:356] "Generic (PLEG): container finished" podID="119c46c3-de88-417e-86a2-e8c808b1e339" containerID="22b75812ebfb580867685902bbcbae7ce856ecf9744501cdc301652f23344363" exitCode=0 Jul 26 16:46:47.874578 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.874406 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" Jul 26 16:46:47.874578 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.874450 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" event={"ID":"119c46c3-de88-417e-86a2-e8c808b1e339","Type":"ContainerDied","Data":"22b75812ebfb580867685902bbcbae7ce856ecf9744501cdc301652f23344363"} Jul 26 16:46:47.874578 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.874510 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788" event={"ID":"119c46c3-de88-417e-86a2-e8c808b1e339","Type":"ContainerDied","Data":"4f2d5c12092e1b04f918bcd12b08dff2786f79497b7de9e946f7103c25ab3989"} Jul 26 16:46:47.874578 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.874543 2563 scope.go:117] "RemoveContainer" containerID="22b75812ebfb580867685902bbcbae7ce856ecf9744501cdc301652f23344363" Jul 26 16:46:47.885074 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.885056 2563 scope.go:117] "RemoveContainer" containerID="872f7e9d6b6f390455fb43b7634c1679395cf77452dd9ad4dd727cf567b826c3" Jul 26 16:46:47.893348 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.893330 2563 scope.go:117] "RemoveContainer" containerID="22b75812ebfb580867685902bbcbae7ce856ecf9744501cdc301652f23344363" Jul 26 16:46:47.893602 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:46:47.893585 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b75812ebfb580867685902bbcbae7ce856ecf9744501cdc301652f23344363\": container with ID starting with 22b75812ebfb580867685902bbcbae7ce856ecf9744501cdc301652f23344363 not found: ID does not exist" containerID="22b75812ebfb580867685902bbcbae7ce856ecf9744501cdc301652f23344363" Jul 26 16:46:47.893650 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.893610 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b75812ebfb580867685902bbcbae7ce856ecf9744501cdc301652f23344363"} err="failed to get container status \"22b75812ebfb580867685902bbcbae7ce856ecf9744501cdc301652f23344363\": rpc error: code = NotFound desc = could not find container \"22b75812ebfb580867685902bbcbae7ce856ecf9744501cdc301652f23344363\": container with ID starting with 22b75812ebfb580867685902bbcbae7ce856ecf9744501cdc301652f23344363 not found: ID does not exist" Jul 26 16:46:47.893650 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.893624 2563 scope.go:117] "RemoveContainer" containerID="872f7e9d6b6f390455fb43b7634c1679395cf77452dd9ad4dd727cf567b826c3" Jul 26 16:46:47.893859 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:46:47.893838 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"872f7e9d6b6f390455fb43b7634c1679395cf77452dd9ad4dd727cf567b826c3\": container with ID starting with 872f7e9d6b6f390455fb43b7634c1679395cf77452dd9ad4dd727cf567b826c3 not found: ID does not exist" containerID="872f7e9d6b6f390455fb43b7634c1679395cf77452dd9ad4dd727cf567b826c3" Jul 26 16:46:47.893924 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.893870 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"872f7e9d6b6f390455fb43b7634c1679395cf77452dd9ad4dd727cf567b826c3"} err="failed to get container status \"872f7e9d6b6f390455fb43b7634c1679395cf77452dd9ad4dd727cf567b826c3\": rpc error: code = NotFound desc = could not find container \"872f7e9d6b6f390455fb43b7634c1679395cf77452dd9ad4dd727cf567b826c3\": container with ID starting with 872f7e9d6b6f390455fb43b7634c1679395cf77452dd9ad4dd727cf567b826c3 not found: ID does not exist" Jul 26 16:46:47.905507 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.905473 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788"] Jul 26 16:46:47.913098 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:47.913074 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e38070-openshift-default-545f994bc6-hm788"] Jul 26 16:46:48.890279 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:46:48.890240 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="119c46c3-de88-417e-86a2-e8c808b1e339" path="/var/lib/kubelet/pods/119c46c3-de88-417e-86a2-e8c808b1e339/volumes" Jul 26 16:47:04.178901 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:04.178864 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx"] Jul 26 16:47:04.179478 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:04.179093 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" podUID="2dc6a646-d246-4596-a570-089ed4b2771a" containerName="istio-proxy" containerID="cri-o://12ae5636373a792b7cc13b19952f5847f81f3870d4c640ee74520290532a8349" gracePeriod=30 Jul 26 16:47:05.054960 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.054925 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r"] Jul 26 16:47:05.055322 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.055309 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41a1a527-6b02-44f0-be04-dd1ac98a60b2" containerName="maas-api" Jul 26 16:47:05.055374 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.055324 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="41a1a527-6b02-44f0-be04-dd1ac98a60b2" containerName="maas-api" Jul 26 16:47:05.055374 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.055341 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5668e27-2f2d-4416-9aff-03a2dece78f2" containerName="revoke-keys" Jul 26 16:47:05.055374 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.055348 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5668e27-2f2d-4416-9aff-03a2dece78f2" containerName="revoke-keys" Jul 26 16:47:05.055374 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.055355 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="119c46c3-de88-417e-86a2-e8c808b1e339" containerName="istio-proxy" Jul 26 16:47:05.055374 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.055362 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="119c46c3-de88-417e-86a2-e8c808b1e339" containerName="istio-proxy" Jul 26 16:47:05.055374 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.055370 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="119c46c3-de88-417e-86a2-e8c808b1e339" containerName="istio-proxy" Jul 26 16:47:05.055374 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.055375 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="119c46c3-de88-417e-86a2-e8c808b1e339" containerName="istio-proxy" Jul 26 16:47:05.055600 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.055433 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="119c46c3-de88-417e-86a2-e8c808b1e339" containerName="istio-proxy" Jul 26 16:47:05.055600 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.055440 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5668e27-2f2d-4416-9aff-03a2dece78f2" containerName="revoke-keys" Jul 26 16:47:05.055600 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.055447 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="41a1a527-6b02-44f0-be04-dd1ac98a60b2" containerName="maas-api" Jul 26 16:47:05.058696 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.058673 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.061732 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.061711 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5425d5-openshift-default-dockercfg-ggvqp\"" Jul 26 16:47:05.073076 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.073050 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r"] Jul 26 16:47:05.144618 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.144580 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43661c45-ce87-45af-a5c7-dd35b0471390-istiod-ca-cert\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.144618 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.144622 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9sdd\" (UniqueName: \"kubernetes.io/projected/43661c45-ce87-45af-a5c7-dd35b0471390-kube-api-access-x9sdd\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.144826 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.144701 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-workload-socket\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.144826 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.144726 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43661c45-ce87-45af-a5c7-dd35b0471390-istio-podinfo\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.144904 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.144827 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43661c45-ce87-45af-a5c7-dd35b0471390-istio-token\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.144949 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.144902 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-workload-certs\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.145002 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.144964 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-istio-data\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.145043 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.145031 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-credential-socket\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.145090 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.145058 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-istio-envoy\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.245906 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.245861 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43661c45-ce87-45af-a5c7-dd35b0471390-istio-token\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.246400 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.245924 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-workload-certs\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.246400 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.245973 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-istio-data\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.246400 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.246019 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-credential-socket\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.246400 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.246041 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-istio-envoy\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.246400 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.246065 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43661c45-ce87-45af-a5c7-dd35b0471390-istiod-ca-cert\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.246400 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.246099 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9sdd\" (UniqueName: \"kubernetes.io/projected/43661c45-ce87-45af-a5c7-dd35b0471390-kube-api-access-x9sdd\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.246400 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.246164 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-workload-socket\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.246400 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.246185 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43661c45-ce87-45af-a5c7-dd35b0471390-istio-podinfo\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.246809 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.246452 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-workload-certs\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.246809 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.246543 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-istio-data\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.246809 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.246601 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-credential-socket\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.246809 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.246664 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-workload-socket\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.247033 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.247007 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43661c45-ce87-45af-a5c7-dd35b0471390-istiod-ca-cert\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.248498 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.248474 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-istio-envoy\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.248632 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.248615 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43661c45-ce87-45af-a5c7-dd35b0471390-istio-podinfo\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.254563 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.254538 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43661c45-ce87-45af-a5c7-dd35b0471390-istio-token\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.254719 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.254699 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9sdd\" (UniqueName: \"kubernetes.io/projected/43661c45-ce87-45af-a5c7-dd35b0471390-kube-api-access-x9sdd\") pod \"e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.368993 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.368897 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:05.505763 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.505647 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r"] Jul 26 16:47:05.508430 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:47:05.508401 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43661c45_ce87_45af_a5c7_dd35b0471390.slice/crio-e5c8798a1cf21c4e666e0b2dbaabf4d5c05d20a2659dd61037693db628db046a WatchSource:0}: Error finding container e5c8798a1cf21c4e666e0b2dbaabf4d5c05d20a2659dd61037693db628db046a: Status 404 returned error can't find the container with id e5c8798a1cf21c4e666e0b2dbaabf4d5c05d20a2659dd61037693db628db046a Jul 26 16:47:05.510495 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.510464 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:47:05.510589 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.510528 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:47:05.510589 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.510556 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:47:05.980257 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.980195 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" event={"ID":"43661c45-ce87-45af-a5c7-dd35b0471390","Type":"ContainerStarted","Data":"e3c7643100b4d4d3b64184dfc1f08665912b6d449586ae39ec5e567a02561609"} Jul 26 16:47:05.980475 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:05.980315 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" event={"ID":"43661c45-ce87-45af-a5c7-dd35b0471390","Type":"ContainerStarted","Data":"e5c8798a1cf21c4e666e0b2dbaabf4d5c05d20a2659dd61037693db628db046a"} Jul 26 16:47:06.004579 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:06.004524 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" podStartSLOduration=1.004510362 podStartE2EDuration="1.004510362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:47:06.000927465 +0000 UTC m=+2299.582014037" watchObservedRunningTime="2026-07-26 16:47:06.004510362 +0000 UTC m=+2299.585596932" Jul 26 16:47:06.369638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:06.369598 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:06.375496 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:06.375469 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:06.985258 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:06.985208 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:06.986208 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:06.986180 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:09.437625 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.437552 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:47:09.482899 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.482864 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc6a646-d246-4596-a570-089ed4b2771a-istio-podinfo\") pod \"2dc6a646-d246-4596-a570-089ed4b2771a\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " Jul 26 16:47:09.483087 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.482958 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-workload-certs\") pod \"2dc6a646-d246-4596-a570-089ed4b2771a\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " Jul 26 16:47:09.483087 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.482988 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srm7s\" (UniqueName: \"kubernetes.io/projected/2dc6a646-d246-4596-a570-089ed4b2771a-kube-api-access-srm7s\") pod \"2dc6a646-d246-4596-a570-089ed4b2771a\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " Jul 26 16:47:09.483087 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.483025 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-istio-envoy\") pod \"2dc6a646-d246-4596-a570-089ed4b2771a\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " Jul 26 16:47:09.483087 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.483057 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc6a646-d246-4596-a570-089ed4b2771a-istiod-ca-cert\") pod \"2dc6a646-d246-4596-a570-089ed4b2771a\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " Jul 26 16:47:09.483087 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.483082 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-credential-socket\") pod \"2dc6a646-d246-4596-a570-089ed4b2771a\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " Jul 26 16:47:09.483394 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.483113 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-istio-data\") pod \"2dc6a646-d246-4596-a570-089ed4b2771a\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " Jul 26 16:47:09.483394 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.483158 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc6a646-d246-4596-a570-089ed4b2771a-istio-token\") pod \"2dc6a646-d246-4596-a570-089ed4b2771a\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " Jul 26 16:47:09.483394 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.483267 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-workload-socket\") pod \"2dc6a646-d246-4596-a570-089ed4b2771a\" (UID: \"2dc6a646-d246-4596-a570-089ed4b2771a\") " Jul 26 16:47:09.483394 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.483293 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2dc6a646-d246-4596-a570-089ed4b2771a" (UID: "2dc6a646-d246-4596-a570-089ed4b2771a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:47:09.483589 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.483504 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc6a646-d246-4596-a570-089ed4b2771a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2dc6a646-d246-4596-a570-089ed4b2771a" (UID: "2dc6a646-d246-4596-a570-089ed4b2771a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:47:09.483589 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.483516 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2dc6a646-d246-4596-a570-089ed4b2771a" (UID: "2dc6a646-d246-4596-a570-089ed4b2771a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:47:09.483700 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.483603 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-workload-certs\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:47:09.483700 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.483620 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc6a646-d246-4596-a570-089ed4b2771a-istiod-ca-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:47:09.483700 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.483633 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-credential-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:47:09.483848 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.483754 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2dc6a646-d246-4596-a570-089ed4b2771a" (UID: "2dc6a646-d246-4596-a570-089ed4b2771a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:47:09.484192 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.484165 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-istio-data" (OuterVolumeSpecName: "istio-data") pod "2dc6a646-d246-4596-a570-089ed4b2771a" (UID: "2dc6a646-d246-4596-a570-089ed4b2771a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:47:09.485462 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.485432 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc6a646-d246-4596-a570-089ed4b2771a-kube-api-access-srm7s" (OuterVolumeSpecName: "kube-api-access-srm7s") pod "2dc6a646-d246-4596-a570-089ed4b2771a" (UID: "2dc6a646-d246-4596-a570-089ed4b2771a"). InnerVolumeSpecName "kube-api-access-srm7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:47:09.485666 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.485644 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dc6a646-d246-4596-a570-089ed4b2771a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2dc6a646-d246-4596-a570-089ed4b2771a" (UID: "2dc6a646-d246-4596-a570-089ed4b2771a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:47:09.485828 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.485806 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc6a646-d246-4596-a570-089ed4b2771a-istio-token" (OuterVolumeSpecName: "istio-token") pod "2dc6a646-d246-4596-a570-089ed4b2771a" (UID: "2dc6a646-d246-4596-a570-089ed4b2771a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:47:09.485900 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.485864 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2dc6a646-d246-4596-a570-089ed4b2771a" (UID: "2dc6a646-d246-4596-a570-089ed4b2771a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:47:09.584784 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.584751 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srm7s\" (UniqueName: \"kubernetes.io/projected/2dc6a646-d246-4596-a570-089ed4b2771a-kube-api-access-srm7s\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:47:09.584784 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.584781 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-istio-envoy\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:47:09.584784 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.584791 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-istio-data\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:47:09.585014 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.584798 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc6a646-d246-4596-a570-089ed4b2771a-istio-token\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:47:09.585014 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.584806 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6a646-d246-4596-a570-089ed4b2771a-workload-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:47:09.585014 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:09.584813 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc6a646-d246-4596-a570-089ed4b2771a-istio-podinfo\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:47:10.004597 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:10.004557 2563 generic.go:356] "Generic (PLEG): container finished" podID="2dc6a646-d246-4596-a570-089ed4b2771a" containerID="12ae5636373a792b7cc13b19952f5847f81f3870d4c640ee74520290532a8349" exitCode=0 Jul 26 16:47:10.004778 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:10.004616 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" event={"ID":"2dc6a646-d246-4596-a570-089ed4b2771a","Type":"ContainerDied","Data":"12ae5636373a792b7cc13b19952f5847f81f3870d4c640ee74520290532a8349"} Jul 26 16:47:10.004778 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:10.004619 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" Jul 26 16:47:10.004778 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:10.004701 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx" event={"ID":"2dc6a646-d246-4596-a570-089ed4b2771a","Type":"ContainerDied","Data":"ffe787b16557cfeb0acd0ab7fa2ea97eb32e4943055aeb09b5da0965fbc7ecf6"} Jul 26 16:47:10.004778 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:10.004735 2563 scope.go:117] "RemoveContainer" containerID="12ae5636373a792b7cc13b19952f5847f81f3870d4c640ee74520290532a8349" Jul 26 16:47:10.015006 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:10.014984 2563 scope.go:117] "RemoveContainer" containerID="dff37e89c3d474607fa3f75fbac2a9f5a8102e4cfad0d61e38a658ba1306fdd3" Jul 26 16:47:10.023738 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:10.023715 2563 scope.go:117] "RemoveContainer" containerID="12ae5636373a792b7cc13b19952f5847f81f3870d4c640ee74520290532a8349" Jul 26 16:47:10.024014 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:47:10.023996 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ae5636373a792b7cc13b19952f5847f81f3870d4c640ee74520290532a8349\": container with ID starting with 12ae5636373a792b7cc13b19952f5847f81f3870d4c640ee74520290532a8349 not found: ID does not exist" containerID="12ae5636373a792b7cc13b19952f5847f81f3870d4c640ee74520290532a8349" Jul 26 16:47:10.024072 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:10.024024 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ae5636373a792b7cc13b19952f5847f81f3870d4c640ee74520290532a8349"} err="failed to get container status \"12ae5636373a792b7cc13b19952f5847f81f3870d4c640ee74520290532a8349\": rpc error: code = NotFound desc = could not find container \"12ae5636373a792b7cc13b19952f5847f81f3870d4c640ee74520290532a8349\": container with ID starting with 12ae5636373a792b7cc13b19952f5847f81f3870d4c640ee74520290532a8349 not found: ID does not exist" Jul 26 16:47:10.024072 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:10.024040 2563 scope.go:117] "RemoveContainer" containerID="dff37e89c3d474607fa3f75fbac2a9f5a8102e4cfad0d61e38a658ba1306fdd3" Jul 26 16:47:10.024278 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:47:10.024263 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff37e89c3d474607fa3f75fbac2a9f5a8102e4cfad0d61e38a658ba1306fdd3\": container with ID starting with dff37e89c3d474607fa3f75fbac2a9f5a8102e4cfad0d61e38a658ba1306fdd3 not found: ID does not exist" containerID="dff37e89c3d474607fa3f75fbac2a9f5a8102e4cfad0d61e38a658ba1306fdd3" Jul 26 16:47:10.024338 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:10.024283 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff37e89c3d474607fa3f75fbac2a9f5a8102e4cfad0d61e38a658ba1306fdd3"} err="failed to get container status \"dff37e89c3d474607fa3f75fbac2a9f5a8102e4cfad0d61e38a658ba1306fdd3\": rpc error: code = NotFound desc = could not find container \"dff37e89c3d474607fa3f75fbac2a9f5a8102e4cfad0d61e38a658ba1306fdd3\": container with ID starting with dff37e89c3d474607fa3f75fbac2a9f5a8102e4cfad0d61e38a658ba1306fdd3 not found: ID does not exist" Jul 26 16:47:10.033557 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:10.033529 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx"] Jul 26 16:47:10.042539 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:10.042513 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6a9b62-openshift-default-574595d44-tqmjx"] Jul 26 16:47:10.891715 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:10.891677 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dc6a646-d246-4596-a570-089ed4b2771a" path="/var/lib/kubelet/pods/2dc6a646-d246-4596-a570-089ed4b2771a/volumes" Jul 26 16:47:13.140407 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.140369 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6"] Jul 26 16:47:13.140812 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.140737 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dc6a646-d246-4596-a570-089ed4b2771a" containerName="istio-proxy" Jul 26 16:47:13.140812 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.140749 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dc6a646-d246-4596-a570-089ed4b2771a" containerName="istio-proxy" Jul 26 16:47:13.140812 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.140763 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dc6a646-d246-4596-a570-089ed4b2771a" containerName="istio-proxy" Jul 26 16:47:13.140812 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.140769 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dc6a646-d246-4596-a570-089ed4b2771a" containerName="istio-proxy" Jul 26 16:47:13.141036 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.140874 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dc6a646-d246-4596-a570-089ed4b2771a" containerName="istio-proxy" Jul 26 16:47:13.141036 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.140889 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="119c46c3-de88-417e-86a2-e8c808b1e339" containerName="istio-proxy" Jul 26 16:47:13.141036 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.140899 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dc6a646-d246-4596-a570-089ed4b2771a" containerName="istio-proxy" Jul 26 16:47:13.145670 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.145647 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.148327 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.148308 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ddd8f8-openshift-default-dockercfg-hxfqd\"" Jul 26 16:47:13.158373 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.158349 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6"] Jul 26 16:47:13.214284 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.214247 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istiod-ca-cert\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.214485 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.214311 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-workload-certs\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.214485 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.214339 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-data\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.214485 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.214409 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-token\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.214485 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.214443 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-credential-socket\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.214485 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.214478 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-podinfo\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.214687 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.214542 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lww2z\" (UniqueName: \"kubernetes.io/projected/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-kube-api-access-lww2z\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.214687 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.214577 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-workload-socket\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.214687 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.214599 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-envoy\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.315255 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.315201 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-token\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.315451 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.315267 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-credential-socket\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.315451 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.315307 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-podinfo\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.315451 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.315341 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lww2z\" (UniqueName: \"kubernetes.io/projected/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-kube-api-access-lww2z\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.315451 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.315377 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-workload-socket\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.315714 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.315683 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-envoy\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.315797 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.315688 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-credential-socket\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.315797 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.315702 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-workload-socket\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.315907 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.315808 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istiod-ca-cert\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.315955 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.315923 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-workload-certs\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.316012 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.315959 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-data\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.316255 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.316212 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-workload-certs\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.316454 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.316250 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-data\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.316679 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.316654 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istiod-ca-cert\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.318165 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.318145 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-envoy\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.318421 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.318396 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-podinfo\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.323761 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.323739 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-token\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.323954 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.323931 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lww2z\" (UniqueName: \"kubernetes.io/projected/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-kube-api-access-lww2z\") pod \"e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.456623 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.456530 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:13.594760 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.594729 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6"] Jul 26 16:47:13.597019 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:47:13.596987 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0342e2_e3d5_4de0_95c2_911ef67ac2b6.slice/crio-03430dd380122b3198ac2f0a9b20791912538bafeb37b909593423c343173469 WatchSource:0}: Error finding container 03430dd380122b3198ac2f0a9b20791912538bafeb37b909593423c343173469: Status 404 returned error can't find the container with id 03430dd380122b3198ac2f0a9b20791912538bafeb37b909593423c343173469 Jul 26 16:47:13.599256 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.599210 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:47:13.599363 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.599308 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:47:13.599363 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:13.599352 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:47:14.034070 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:14.034023 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" event={"ID":"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6","Type":"ContainerStarted","Data":"c9d8087f45d26151cab30dc4fce5c147e05b600f26726c412e639ce518cedfdd"} Jul 26 16:47:14.034310 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:14.034086 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" event={"ID":"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6","Type":"ContainerStarted","Data":"03430dd380122b3198ac2f0a9b20791912538bafeb37b909593423c343173469"} Jul 26 16:47:14.054607 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:14.054517 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" podStartSLOduration=1.054492123 podStartE2EDuration="1.054492123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:47:14.053107699 +0000 UTC m=+2307.634194296" watchObservedRunningTime="2026-07-26 16:47:14.054492123 +0000 UTC m=+2307.635578743" Jul 26 16:47:14.456771 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:14.456671 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:14.462200 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:14.462171 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:15.039889 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:15.039859 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:15.040849 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:15.040828 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:16.027099 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:16.027064 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s"] Jul 26 16:47:16.030875 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:16.030857 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" Jul 26 16:47:16.033641 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:16.033616 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ddd8f8-serving-cert\"" Jul 26 16:47:16.046629 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:16.046591 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s"] Jul 26 16:47:16.142700 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:16.142663 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9de1daa7-e7fd-4649-8cc3-280550ba8c44-maas-api-tls\") pod \"maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s\" (UID: \"9de1daa7-e7fd-4649-8cc3-280550ba8c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" Jul 26 16:47:16.142899 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:16.142766 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9de1daa7-e7fd-4649-8cc3-280550ba8c44-openshift-service-ca\") pod \"maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s\" (UID: \"9de1daa7-e7fd-4649-8cc3-280550ba8c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" Jul 26 16:47:16.142899 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:16.142797 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6r2w\" (UniqueName: \"kubernetes.io/projected/9de1daa7-e7fd-4649-8cc3-280550ba8c44-kube-api-access-l6r2w\") pod \"maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s\" (UID: \"9de1daa7-e7fd-4649-8cc3-280550ba8c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" Jul 26 16:47:16.243483 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:16.243446 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9de1daa7-e7fd-4649-8cc3-280550ba8c44-openshift-service-ca\") pod \"maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s\" (UID: \"9de1daa7-e7fd-4649-8cc3-280550ba8c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" Jul 26 16:47:16.243483 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:16.243491 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6r2w\" (UniqueName: \"kubernetes.io/projected/9de1daa7-e7fd-4649-8cc3-280550ba8c44-kube-api-access-l6r2w\") pod \"maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s\" (UID: \"9de1daa7-e7fd-4649-8cc3-280550ba8c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" Jul 26 16:47:16.243713 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:16.243563 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9de1daa7-e7fd-4649-8cc3-280550ba8c44-maas-api-tls\") pod \"maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s\" (UID: \"9de1daa7-e7fd-4649-8cc3-280550ba8c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" Jul 26 16:47:16.244202 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:16.244174 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9de1daa7-e7fd-4649-8cc3-280550ba8c44-openshift-service-ca\") pod \"maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s\" (UID: \"9de1daa7-e7fd-4649-8cc3-280550ba8c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" Jul 26 16:47:16.246057 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:16.246025 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9de1daa7-e7fd-4649-8cc3-280550ba8c44-maas-api-tls\") pod \"maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s\" (UID: \"9de1daa7-e7fd-4649-8cc3-280550ba8c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" Jul 26 16:47:16.254744 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:16.254712 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6r2w\" (UniqueName: \"kubernetes.io/projected/9de1daa7-e7fd-4649-8cc3-280550ba8c44-kube-api-access-l6r2w\") pod \"maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s\" (UID: \"9de1daa7-e7fd-4649-8cc3-280550ba8c44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" Jul 26 16:47:16.342256 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:16.342144 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" Jul 26 16:47:16.487796 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:16.487755 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s"] Jul 26 16:47:16.506289 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:47:16.504811 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de1daa7_e7fd_4649_8cc3_280550ba8c44.slice/crio-a65ec537a88710b5e657e0ce80f454fddf6cb9dcecc8b25b242e14291e85ebf0 WatchSource:0}: Error finding container a65ec537a88710b5e657e0ce80f454fddf6cb9dcecc8b25b242e14291e85ebf0: Status 404 returned error can't find the container with id a65ec537a88710b5e657e0ce80f454fddf6cb9dcecc8b25b242e14291e85ebf0 Jul 26 16:47:16.507852 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:16.507815 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 16:47:17.075640 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:17.075421 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" event={"ID":"9de1daa7-e7fd-4649-8cc3-280550ba8c44","Type":"ContainerStarted","Data":"0acbe453a26121d5b409b6c1b8ab67ac2d6ea0bbc7c03baa0f9064cfd000b3b3"} Jul 26 16:47:17.075640 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:17.075494 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" event={"ID":"9de1daa7-e7fd-4649-8cc3-280550ba8c44","Type":"ContainerStarted","Data":"a65ec537a88710b5e657e0ce80f454fddf6cb9dcecc8b25b242e14291e85ebf0"} Jul 26 16:47:17.075640 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:17.075540 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" Jul 26 16:47:17.107870 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:17.107814 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" podStartSLOduration=0.790577481 podStartE2EDuration="1.107797857s" totalImagesPullingTime="317.220376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:47:17.10388532 +0000 UTC m=+2310.684971890" watchObservedRunningTime="2026-07-26 16:47:17.107797857 +0000 UTC m=+2310.688884485" Jul 26 16:47:23.027389 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:23.027343 2563 patch_prober.go:28] interesting pod/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 26 16:47:23.027785 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:23.027405 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" podUID="43661c45-ce87-45af-a5c7-dd35b0471390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 26 16:47:24.094827 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:24.094787 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" Jul 26 16:47:24.120894 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:24.120860 2563 generic.go:356] "Generic (PLEG): container finished" podID="43661c45-ce87-45af-a5c7-dd35b0471390" containerID="e3c7643100b4d4d3b64184dfc1f08665912b6d449586ae39ec5e567a02561609" exitCode=0 Jul 26 16:47:24.121072 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:24.120938 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" event={"ID":"43661c45-ce87-45af-a5c7-dd35b0471390","Type":"ContainerDied","Data":"e3c7643100b4d4d3b64184dfc1f08665912b6d449586ae39ec5e567a02561609"} Jul 26 16:47:24.121438 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:24.121409 2563 scope.go:117] "RemoveContainer" containerID="e3c7643100b4d4d3b64184dfc1f08665912b6d449586ae39ec5e567a02561609" Jul 26 16:47:24.122624 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:24.122584 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:47:24.122723 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:24.122668 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:47:24.122785 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:24.122720 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:47:25.128320 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:25.128279 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" event={"ID":"43661c45-ce87-45af-a5c7-dd35b0471390","Type":"ContainerStarted","Data":"7c419a80d17b1e981c00c3ee20b81f9808b2ffe0cf369e9fbcf37a47df052e3d"} Jul 26 16:47:25.369984 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:25.369938 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:25.370252 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:25.370201 2563 patch_prober.go:28] interesting pod/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 26 16:47:25.370345 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:25.370299 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" podUID="43661c45-ce87-45af-a5c7-dd35b0471390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 26 16:47:26.370057 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:26.370006 2563 patch_prober.go:28] interesting pod/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 26 16:47:26.370592 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:26.370080 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" podUID="43661c45-ce87-45af-a5c7-dd35b0471390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 26 16:47:27.369953 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:27.369907 2563 patch_prober.go:28] interesting pod/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 26 16:47:27.370184 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:27.369978 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" podUID="43661c45-ce87-45af-a5c7-dd35b0471390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 26 16:47:28.370100 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:28.370060 2563 patch_prober.go:28] interesting pod/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 26 16:47:28.370503 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:28.370461 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" podUID="43661c45-ce87-45af-a5c7-dd35b0471390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 26 16:47:29.373146 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:29.373113 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:29.373690 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:29.373510 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:29.374325 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:29.374303 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:47:31.081434 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:31.081387 2563 patch_prober.go:28] interesting pod/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 26 16:47:31.081852 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:31.081487 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" podUID="cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 26 16:47:32.171152 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:32.171117 2563 generic.go:356] "Generic (PLEG): container finished" podID="cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" containerID="c9d8087f45d26151cab30dc4fce5c147e05b600f26726c412e639ce518cedfdd" exitCode=0 Jul 26 16:47:32.171638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:32.171165 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" event={"ID":"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6","Type":"ContainerDied","Data":"c9d8087f45d26151cab30dc4fce5c147e05b600f26726c412e639ce518cedfdd"} Jul 26 16:47:32.171638 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:32.171545 2563 scope.go:117] "RemoveContainer" containerID="c9d8087f45d26151cab30dc4fce5c147e05b600f26726c412e639ce518cedfdd" Jul 26 16:47:32.172556 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:32.172528 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:47:32.172624 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:32.172595 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:47:32.172666 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:32.172627 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:47:33.178779 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:33.178734 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" event={"ID":"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6","Type":"ContainerStarted","Data":"76875245fb090327b3369662868235a7dbd2ef32590b6b06e3425798678d29cf"} Jul 26 16:47:33.457011 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:33.456903 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:33.457192 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:33.457041 2563 patch_prober.go:28] interesting pod/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 26 16:47:33.457192 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:33.457095 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" podUID="cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 26 16:47:34.457392 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:34.457346 2563 patch_prober.go:28] interesting pod/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 26 16:47:34.457949 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:34.457426 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" podUID="cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 26 16:47:35.458007 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:35.457963 2563 patch_prober.go:28] interesting pod/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 26 16:47:35.458667 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:35.458037 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" podUID="cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 26 16:47:36.457907 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:36.457862 2563 patch_prober.go:28] interesting pod/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 26 16:47:36.458131 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:36.457938 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" podUID="cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 26 16:47:37.460488 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:37.460456 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:37.460981 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:37.460847 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:37.461573 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:37.461553 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:47:55.016795 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:55.016756 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r"] Jul 26 16:47:55.017685 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:55.017649 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" podUID="43661c45-ce87-45af-a5c7-dd35b0471390" containerName="istio-proxy" containerID="cri-o://7c419a80d17b1e981c00c3ee20b81f9808b2ffe0cf369e9fbcf37a47df052e3d" gracePeriod=30 Jul 26 16:47:59.376037 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:59.375996 2563 patch_prober.go:28] interesting pod/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 16:47:59.376459 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:47:59.376079 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" podUID="43661c45-ce87-45af-a5c7-dd35b0471390" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 16:48:00.275648 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.275624 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:48:00.328495 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.328410 2563 generic.go:356] "Generic (PLEG): container finished" podID="43661c45-ce87-45af-a5c7-dd35b0471390" containerID="7c419a80d17b1e981c00c3ee20b81f9808b2ffe0cf369e9fbcf37a47df052e3d" exitCode=0 Jul 26 16:48:00.328495 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.328471 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" Jul 26 16:48:00.328664 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.328472 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" event={"ID":"43661c45-ce87-45af-a5c7-dd35b0471390","Type":"ContainerDied","Data":"7c419a80d17b1e981c00c3ee20b81f9808b2ffe0cf369e9fbcf37a47df052e3d"} Jul 26 16:48:00.328664 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.328598 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r" event={"ID":"43661c45-ce87-45af-a5c7-dd35b0471390","Type":"ContainerDied","Data":"e5c8798a1cf21c4e666e0b2dbaabf4d5c05d20a2659dd61037693db628db046a"} Jul 26 16:48:00.328664 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.328620 2563 scope.go:117] "RemoveContainer" containerID="7c419a80d17b1e981c00c3ee20b81f9808b2ffe0cf369e9fbcf37a47df052e3d" Jul 26 16:48:00.337752 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.337732 2563 scope.go:117] "RemoveContainer" containerID="e3c7643100b4d4d3b64184dfc1f08665912b6d449586ae39ec5e567a02561609" Jul 26 16:48:00.345626 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.345609 2563 scope.go:117] "RemoveContainer" containerID="7c419a80d17b1e981c00c3ee20b81f9808b2ffe0cf369e9fbcf37a47df052e3d" Jul 26 16:48:00.345878 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:48:00.345857 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c419a80d17b1e981c00c3ee20b81f9808b2ffe0cf369e9fbcf37a47df052e3d\": container with ID starting with 7c419a80d17b1e981c00c3ee20b81f9808b2ffe0cf369e9fbcf37a47df052e3d not found: ID does not exist" containerID="7c419a80d17b1e981c00c3ee20b81f9808b2ffe0cf369e9fbcf37a47df052e3d" Jul 26 16:48:00.345931 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.345893 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c419a80d17b1e981c00c3ee20b81f9808b2ffe0cf369e9fbcf37a47df052e3d"} err="failed to get container status \"7c419a80d17b1e981c00c3ee20b81f9808b2ffe0cf369e9fbcf37a47df052e3d\": rpc error: code = NotFound desc = could not find container \"7c419a80d17b1e981c00c3ee20b81f9808b2ffe0cf369e9fbcf37a47df052e3d\": container with ID starting with 7c419a80d17b1e981c00c3ee20b81f9808b2ffe0cf369e9fbcf37a47df052e3d not found: ID does not exist" Jul 26 16:48:00.345931 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.345911 2563 scope.go:117] "RemoveContainer" containerID="e3c7643100b4d4d3b64184dfc1f08665912b6d449586ae39ec5e567a02561609" Jul 26 16:48:00.346156 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:48:00.346138 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c7643100b4d4d3b64184dfc1f08665912b6d449586ae39ec5e567a02561609\": container with ID starting with e3c7643100b4d4d3b64184dfc1f08665912b6d449586ae39ec5e567a02561609 not found: ID does not exist" containerID="e3c7643100b4d4d3b64184dfc1f08665912b6d449586ae39ec5e567a02561609" Jul 26 16:48:00.346246 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.346165 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c7643100b4d4d3b64184dfc1f08665912b6d449586ae39ec5e567a02561609"} err="failed to get container status \"e3c7643100b4d4d3b64184dfc1f08665912b6d449586ae39ec5e567a02561609\": rpc error: code = NotFound desc = could not find container \"e3c7643100b4d4d3b64184dfc1f08665912b6d449586ae39ec5e567a02561609\": container with ID starting with e3c7643100b4d4d3b64184dfc1f08665912b6d449586ae39ec5e567a02561609 not found: ID does not exist" Jul 26 16:48:00.379567 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.379540 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-workload-certs\") pod \"43661c45-ce87-45af-a5c7-dd35b0471390\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " Jul 26 16:48:00.379914 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.379573 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-istio-data\") pod \"43661c45-ce87-45af-a5c7-dd35b0471390\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " Jul 26 16:48:00.379914 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.379615 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9sdd\" (UniqueName: \"kubernetes.io/projected/43661c45-ce87-45af-a5c7-dd35b0471390-kube-api-access-x9sdd\") pod \"43661c45-ce87-45af-a5c7-dd35b0471390\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " Jul 26 16:48:00.379914 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.379657 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43661c45-ce87-45af-a5c7-dd35b0471390-istiod-ca-cert\") pod \"43661c45-ce87-45af-a5c7-dd35b0471390\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " Jul 26 16:48:00.379914 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.379727 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-workload-socket\") pod \"43661c45-ce87-45af-a5c7-dd35b0471390\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " Jul 26 16:48:00.379914 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.379765 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43661c45-ce87-45af-a5c7-dd35b0471390-istio-token\") pod \"43661c45-ce87-45af-a5c7-dd35b0471390\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " Jul 26 16:48:00.379914 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.379794 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43661c45-ce87-45af-a5c7-dd35b0471390-istio-podinfo\") pod \"43661c45-ce87-45af-a5c7-dd35b0471390\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " Jul 26 16:48:00.379914 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.379845 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "43661c45-ce87-45af-a5c7-dd35b0471390" (UID: "43661c45-ce87-45af-a5c7-dd35b0471390"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:48:00.379914 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.379874 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-istio-envoy\") pod \"43661c45-ce87-45af-a5c7-dd35b0471390\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " Jul 26 16:48:00.380355 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.379941 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-credential-socket\") pod \"43661c45-ce87-45af-a5c7-dd35b0471390\" (UID: \"43661c45-ce87-45af-a5c7-dd35b0471390\") " Jul 26 16:48:00.380355 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.380057 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "43661c45-ce87-45af-a5c7-dd35b0471390" (UID: "43661c45-ce87-45af-a5c7-dd35b0471390"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:48:00.380355 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.380116 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43661c45-ce87-45af-a5c7-dd35b0471390-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "43661c45-ce87-45af-a5c7-dd35b0471390" (UID: "43661c45-ce87-45af-a5c7-dd35b0471390"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:48:00.380355 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.380333 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "43661c45-ce87-45af-a5c7-dd35b0471390" (UID: "43661c45-ce87-45af-a5c7-dd35b0471390"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:48:00.380576 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.380424 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-workload-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:00.380576 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.380444 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-credential-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:00.380576 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.380459 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-workload-certs\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:00.380576 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.380471 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43661c45-ce87-45af-a5c7-dd35b0471390-istiod-ca-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:00.380576 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.380567 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-istio-data" (OuterVolumeSpecName: "istio-data") pod "43661c45-ce87-45af-a5c7-dd35b0471390" (UID: "43661c45-ce87-45af-a5c7-dd35b0471390"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:48:00.382054 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.382033 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43661c45-ce87-45af-a5c7-dd35b0471390-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "43661c45-ce87-45af-a5c7-dd35b0471390" (UID: "43661c45-ce87-45af-a5c7-dd35b0471390"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:48:00.382201 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.382185 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43661c45-ce87-45af-a5c7-dd35b0471390-kube-api-access-x9sdd" (OuterVolumeSpecName: "kube-api-access-x9sdd") pod "43661c45-ce87-45af-a5c7-dd35b0471390" (UID: "43661c45-ce87-45af-a5c7-dd35b0471390"). InnerVolumeSpecName "kube-api-access-x9sdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:48:00.382443 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.382421 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43661c45-ce87-45af-a5c7-dd35b0471390-istio-token" (OuterVolumeSpecName: "istio-token") pod "43661c45-ce87-45af-a5c7-dd35b0471390" (UID: "43661c45-ce87-45af-a5c7-dd35b0471390"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:48:00.382500 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.382455 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "43661c45-ce87-45af-a5c7-dd35b0471390" (UID: "43661c45-ce87-45af-a5c7-dd35b0471390"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:48:00.481206 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.481163 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43661c45-ce87-45af-a5c7-dd35b0471390-istio-token\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:00.481206 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.481197 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43661c45-ce87-45af-a5c7-dd35b0471390-istio-podinfo\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:00.481206 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.481208 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-istio-envoy\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:00.481206 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.481216 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43661c45-ce87-45af-a5c7-dd35b0471390-istio-data\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:00.481507 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.481250 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9sdd\" (UniqueName: \"kubernetes.io/projected/43661c45-ce87-45af-a5c7-dd35b0471390-kube-api-access-x9sdd\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:00.666686 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.666653 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r"] Jul 26 16:48:00.671448 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.671418 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5425d5-openshift-default-7bd5dd78bb-kpp2r"] Jul 26 16:48:00.891066 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:00.890982 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43661c45-ce87-45af-a5c7-dd35b0471390" path="/var/lib/kubelet/pods/43661c45-ce87-45af-a5c7-dd35b0471390/volumes" Jul 26 16:48:10.981924 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:10.981871 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s"] Jul 26 16:48:10.982732 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:10.982164 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" podUID="9de1daa7-e7fd-4649-8cc3-280550ba8c44" containerName="maas-api" containerID="cri-o://0acbe453a26121d5b409b6c1b8ab67ac2d6ea0bbc7c03baa0f9064cfd000b3b3" gracePeriod=30 Jul 26 16:48:11.247713 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.247644 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" Jul 26 16:48:11.374575 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.374540 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9de1daa7-e7fd-4649-8cc3-280550ba8c44-maas-api-tls\") pod \"9de1daa7-e7fd-4649-8cc3-280550ba8c44\" (UID: \"9de1daa7-e7fd-4649-8cc3-280550ba8c44\") " Jul 26 16:48:11.374767 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.374596 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9de1daa7-e7fd-4649-8cc3-280550ba8c44-openshift-service-ca\") pod \"9de1daa7-e7fd-4649-8cc3-280550ba8c44\" (UID: \"9de1daa7-e7fd-4649-8cc3-280550ba8c44\") " Jul 26 16:48:11.374767 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.374685 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6r2w\" (UniqueName: \"kubernetes.io/projected/9de1daa7-e7fd-4649-8cc3-280550ba8c44-kube-api-access-l6r2w\") pod \"9de1daa7-e7fd-4649-8cc3-280550ba8c44\" (UID: \"9de1daa7-e7fd-4649-8cc3-280550ba8c44\") " Jul 26 16:48:11.375029 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.374999 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de1daa7-e7fd-4649-8cc3-280550ba8c44-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9de1daa7-e7fd-4649-8cc3-280550ba8c44" (UID: "9de1daa7-e7fd-4649-8cc3-280550ba8c44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:48:11.376671 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.376647 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9de1daa7-e7fd-4649-8cc3-280550ba8c44-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9de1daa7-e7fd-4649-8cc3-280550ba8c44" (UID: "9de1daa7-e7fd-4649-8cc3-280550ba8c44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:48:11.376735 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.376710 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de1daa7-e7fd-4649-8cc3-280550ba8c44-kube-api-access-l6r2w" (OuterVolumeSpecName: "kube-api-access-l6r2w") pod "9de1daa7-e7fd-4649-8cc3-280550ba8c44" (UID: "9de1daa7-e7fd-4649-8cc3-280550ba8c44"). InnerVolumeSpecName "kube-api-access-l6r2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:48:11.390985 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.390946 2563 generic.go:356] "Generic (PLEG): container finished" podID="9de1daa7-e7fd-4649-8cc3-280550ba8c44" containerID="0acbe453a26121d5b409b6c1b8ab67ac2d6ea0bbc7c03baa0f9064cfd000b3b3" exitCode=0 Jul 26 16:48:11.390985 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.390975 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" event={"ID":"9de1daa7-e7fd-4649-8cc3-280550ba8c44","Type":"ContainerDied","Data":"0acbe453a26121d5b409b6c1b8ab67ac2d6ea0bbc7c03baa0f9064cfd000b3b3"} Jul 26 16:48:11.391183 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.391034 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" event={"ID":"9de1daa7-e7fd-4649-8cc3-280550ba8c44","Type":"ContainerDied","Data":"a65ec537a88710b5e657e0ce80f454fddf6cb9dcecc8b25b242e14291e85ebf0"} Jul 26 16:48:11.391183 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.391050 2563 scope.go:117] "RemoveContainer" containerID="0acbe453a26121d5b409b6c1b8ab67ac2d6ea0bbc7c03baa0f9064cfd000b3b3" Jul 26 16:48:11.391183 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.391031 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s" Jul 26 16:48:11.400109 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.400091 2563 scope.go:117] "RemoveContainer" containerID="0acbe453a26121d5b409b6c1b8ab67ac2d6ea0bbc7c03baa0f9064cfd000b3b3" Jul 26 16:48:11.400424 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:48:11.400401 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0acbe453a26121d5b409b6c1b8ab67ac2d6ea0bbc7c03baa0f9064cfd000b3b3\": container with ID starting with 0acbe453a26121d5b409b6c1b8ab67ac2d6ea0bbc7c03baa0f9064cfd000b3b3 not found: ID does not exist" containerID="0acbe453a26121d5b409b6c1b8ab67ac2d6ea0bbc7c03baa0f9064cfd000b3b3" Jul 26 16:48:11.400501 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.400430 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acbe453a26121d5b409b6c1b8ab67ac2d6ea0bbc7c03baa0f9064cfd000b3b3"} err="failed to get container status \"0acbe453a26121d5b409b6c1b8ab67ac2d6ea0bbc7c03baa0f9064cfd000b3b3\": rpc error: code = NotFound desc = could not find container \"0acbe453a26121d5b409b6c1b8ab67ac2d6ea0bbc7c03baa0f9064cfd000b3b3\": container with ID starting with 0acbe453a26121d5b409b6c1b8ab67ac2d6ea0bbc7c03baa0f9064cfd000b3b3 not found: ID does not exist" Jul 26 16:48:11.413742 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.413713 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s"] Jul 26 16:48:11.418319 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.418293 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ddd8f8-6f6bf9679-q466s"] Jul 26 16:48:11.475754 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.475704 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9de1daa7-e7fd-4649-8cc3-280550ba8c44-openshift-service-ca\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:11.475754 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.475746 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6r2w\" (UniqueName: \"kubernetes.io/projected/9de1daa7-e7fd-4649-8cc3-280550ba8c44-kube-api-access-l6r2w\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:11.475754 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:11.475758 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9de1daa7-e7fd-4649-8cc3-280550ba8c44-maas-api-tls\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:12.892276 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:12.892241 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9de1daa7-e7fd-4649-8cc3-280550ba8c44" path="/var/lib/kubelet/pods/9de1daa7-e7fd-4649-8cc3-280550ba8c44/volumes" Jul 26 16:48:22.145238 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:22.145184 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6"] Jul 26 16:48:22.145798 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:22.145456 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" podUID="cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" containerName="istio-proxy" containerID="cri-o://76875245fb090327b3369662868235a7dbd2ef32590b6b06e3425798678d29cf" gracePeriod=30 Jul 26 16:48:22.463786 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:22.463704 2563 patch_prober.go:28] interesting pod/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 16:48:22.463933 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:22.463792 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" podUID="cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 16:48:23.087734 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.087694 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc"] Jul 26 16:48:23.088079 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.088066 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9de1daa7-e7fd-4649-8cc3-280550ba8c44" containerName="maas-api" Jul 26 16:48:23.088125 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.088082 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9de1daa7-e7fd-4649-8cc3-280550ba8c44" containerName="maas-api" Jul 26 16:48:23.088125 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.088091 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43661c45-ce87-45af-a5c7-dd35b0471390" containerName="istio-proxy" Jul 26 16:48:23.088125 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.088097 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="43661c45-ce87-45af-a5c7-dd35b0471390" containerName="istio-proxy" Jul 26 16:48:23.088250 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.088173 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="43661c45-ce87-45af-a5c7-dd35b0471390" containerName="istio-proxy" Jul 26 16:48:23.088250 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.088182 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9de1daa7-e7fd-4649-8cc3-280550ba8c44" containerName="maas-api" Jul 26 16:48:23.091320 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.091302 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.095189 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.095161 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-1b2072-openshift-default-dockercfg-b2gjr\"" Jul 26 16:48:23.123723 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.123674 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc"] Jul 26 16:48:23.181598 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.181561 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf8v7\" (UniqueName: \"kubernetes.io/projected/a654d489-3ddf-4fd5-b428-f5cba87350c1-kube-api-access-vf8v7\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.182070 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.181608 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-workload-socket\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.182070 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.181627 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-credential-socket\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.182070 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.181649 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a654d489-3ddf-4fd5-b428-f5cba87350c1-istiod-ca-cert\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.182070 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.181737 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-workload-certs\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.182070 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.181801 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-data\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.182070 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.181829 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-token\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.182070 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.181927 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-podinfo\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.182070 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.181962 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-envoy\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.282955 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.282913 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-data\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.282955 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.282954 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-token\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.283203 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.283077 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-podinfo\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.283203 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.283123 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-envoy\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.283203 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.283156 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf8v7\" (UniqueName: \"kubernetes.io/projected/a654d489-3ddf-4fd5-b428-f5cba87350c1-kube-api-access-vf8v7\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.283203 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.283196 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-workload-socket\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.283486 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.283214 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-credential-socket\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.283486 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.283315 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a654d489-3ddf-4fd5-b428-f5cba87350c1-istiod-ca-cert\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.283486 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.283375 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-workload-certs\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.283486 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.283392 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-data\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.283684 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.283534 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-workload-socket\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.283747 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.283725 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-workload-certs\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.283860 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.283840 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-credential-socket\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.284030 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.284013 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a654d489-3ddf-4fd5-b428-f5cba87350c1-istiod-ca-cert\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.285563 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.285538 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-envoy\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.285674 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.285655 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-podinfo\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.291923 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.291898 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf8v7\" (UniqueName: \"kubernetes.io/projected/a654d489-3ddf-4fd5-b428-f5cba87350c1-kube-api-access-vf8v7\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.293713 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.293693 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-token\") pod \"e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.403392 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.403300 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:23.551947 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.551916 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc"] Jul 26 16:48:23.553338 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:48:23.553309 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda654d489_3ddf_4fd5_b428_f5cba87350c1.slice/crio-3ad8d72ffe8f18791dcae8c5b7fd6321ff69838dc7c95576413cb7c0a2055e01 WatchSource:0}: Error finding container 3ad8d72ffe8f18791dcae8c5b7fd6321ff69838dc7c95576413cb7c0a2055e01: Status 404 returned error can't find the container with id 3ad8d72ffe8f18791dcae8c5b7fd6321ff69838dc7c95576413cb7c0a2055e01 Jul 26 16:48:23.555474 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.555443 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:48:23.555544 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.555512 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:48:23.555544 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:23.555540 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:48:24.460868 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:24.460826 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" event={"ID":"a654d489-3ddf-4fd5-b428-f5cba87350c1","Type":"ContainerStarted","Data":"6aa3ee5275119b02877e894ebcd004a82a8b6a1957fd970d808bbda2174b6c54"} Jul 26 16:48:24.461297 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:24.460895 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" event={"ID":"a654d489-3ddf-4fd5-b428-f5cba87350c1","Type":"ContainerStarted","Data":"3ad8d72ffe8f18791dcae8c5b7fd6321ff69838dc7c95576413cb7c0a2055e01"} Jul 26 16:48:24.483964 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:24.483917 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" podStartSLOduration=1.483901377 podStartE2EDuration="1.483901377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:48:24.48375649 +0000 UTC m=+2378.064843062" watchObservedRunningTime="2026-07-26 16:48:24.483901377 +0000 UTC m=+2378.064987947" Jul 26 16:48:25.404183 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:25.404142 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:25.408789 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:25.408765 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:25.467170 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:25.467140 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:25.468159 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:25.468139 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:27.408957 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.408891 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:48:27.480873 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.480842 2563 generic.go:356] "Generic (PLEG): container finished" podID="cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" containerID="76875245fb090327b3369662868235a7dbd2ef32590b6b06e3425798678d29cf" exitCode=0 Jul 26 16:48:27.481065 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.480895 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" Jul 26 16:48:27.481065 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.480923 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" event={"ID":"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6","Type":"ContainerDied","Data":"76875245fb090327b3369662868235a7dbd2ef32590b6b06e3425798678d29cf"} Jul 26 16:48:27.481065 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.480984 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6" event={"ID":"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6","Type":"ContainerDied","Data":"03430dd380122b3198ac2f0a9b20791912538bafeb37b909593423c343173469"} Jul 26 16:48:27.481065 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.481020 2563 scope.go:117] "RemoveContainer" containerID="76875245fb090327b3369662868235a7dbd2ef32590b6b06e3425798678d29cf" Jul 26 16:48:27.490963 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.490943 2563 scope.go:117] "RemoveContainer" containerID="c9d8087f45d26151cab30dc4fce5c147e05b600f26726c412e639ce518cedfdd" Jul 26 16:48:27.499411 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.499386 2563 scope.go:117] "RemoveContainer" containerID="76875245fb090327b3369662868235a7dbd2ef32590b6b06e3425798678d29cf" Jul 26 16:48:27.499689 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:48:27.499670 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76875245fb090327b3369662868235a7dbd2ef32590b6b06e3425798678d29cf\": container with ID starting with 76875245fb090327b3369662868235a7dbd2ef32590b6b06e3425798678d29cf not found: ID does not exist" containerID="76875245fb090327b3369662868235a7dbd2ef32590b6b06e3425798678d29cf" Jul 26 16:48:27.499747 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.499707 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76875245fb090327b3369662868235a7dbd2ef32590b6b06e3425798678d29cf"} err="failed to get container status \"76875245fb090327b3369662868235a7dbd2ef32590b6b06e3425798678d29cf\": rpc error: code = NotFound desc = could not find container \"76875245fb090327b3369662868235a7dbd2ef32590b6b06e3425798678d29cf\": container with ID starting with 76875245fb090327b3369662868235a7dbd2ef32590b6b06e3425798678d29cf not found: ID does not exist" Jul 26 16:48:27.499747 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.499726 2563 scope.go:117] "RemoveContainer" containerID="c9d8087f45d26151cab30dc4fce5c147e05b600f26726c412e639ce518cedfdd" Jul 26 16:48:27.499942 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:48:27.499926 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d8087f45d26151cab30dc4fce5c147e05b600f26726c412e639ce518cedfdd\": container with ID starting with c9d8087f45d26151cab30dc4fce5c147e05b600f26726c412e639ce518cedfdd not found: ID does not exist" containerID="c9d8087f45d26151cab30dc4fce5c147e05b600f26726c412e639ce518cedfdd" Jul 26 16:48:27.499986 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.499953 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d8087f45d26151cab30dc4fce5c147e05b600f26726c412e639ce518cedfdd"} err="failed to get container status \"c9d8087f45d26151cab30dc4fce5c147e05b600f26726c412e639ce518cedfdd\": rpc error: code = NotFound desc = could not find container \"c9d8087f45d26151cab30dc4fce5c147e05b600f26726c412e639ce518cedfdd\": container with ID starting with c9d8087f45d26151cab30dc4fce5c147e05b600f26726c412e639ce518cedfdd not found: ID does not exist" Jul 26 16:48:27.524398 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.524373 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-data\") pod \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " Jul 26 16:48:27.524521 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.524457 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istiod-ca-cert\") pod \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " Jul 26 16:48:27.524521 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.524495 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-token\") pod \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " Jul 26 16:48:27.524601 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.524520 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-podinfo\") pod \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " Jul 26 16:48:27.524601 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.524569 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-credential-socket\") pod \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " Jul 26 16:48:27.524601 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.524598 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-workload-certs\") pod \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " Jul 26 16:48:27.524760 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.524622 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lww2z\" (UniqueName: \"kubernetes.io/projected/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-kube-api-access-lww2z\") pod \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " Jul 26 16:48:27.524924 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.524902 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-envoy\") pod \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " Jul 26 16:48:27.525264 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.524855 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" (UID: "cf0342e2-e3d5-4de0-95c2-911ef67ac2b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:48:27.525264 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.524874 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" (UID: "cf0342e2-e3d5-4de0-95c2-911ef67ac2b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:48:27.525264 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.524942 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-workload-socket\") pod \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\" (UID: \"cf0342e2-e3d5-4de0-95c2-911ef67ac2b6\") " Jul 26 16:48:27.525264 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.525099 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" (UID: "cf0342e2-e3d5-4de0-95c2-911ef67ac2b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:48:27.525264 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.525167 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" (UID: "cf0342e2-e3d5-4de0-95c2-911ef67ac2b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:48:27.525594 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.525381 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-credential-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:27.525594 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.525396 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-workload-certs\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:27.525594 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.525411 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-workload-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:27.525594 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.525423 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istiod-ca-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:27.525594 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.525459 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" (UID: "cf0342e2-e3d5-4de0-95c2-911ef67ac2b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:48:27.527307 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.527283 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-kube-api-access-lww2z" (OuterVolumeSpecName: "kube-api-access-lww2z") pod "cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" (UID: "cf0342e2-e3d5-4de0-95c2-911ef67ac2b6"). InnerVolumeSpecName "kube-api-access-lww2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:48:27.527554 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.527522 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" (UID: "cf0342e2-e3d5-4de0-95c2-911ef67ac2b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:48:27.527661 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.527562 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" (UID: "cf0342e2-e3d5-4de0-95c2-911ef67ac2b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:48:27.527661 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.527628 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" (UID: "cf0342e2-e3d5-4de0-95c2-911ef67ac2b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:48:27.626535 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.626497 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-token\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:27.626535 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.626529 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-podinfo\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:27.626535 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.626538 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lww2z\" (UniqueName: \"kubernetes.io/projected/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-kube-api-access-lww2z\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:27.626781 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.626547 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-envoy\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:27.626781 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.626556 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6-istio-data\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 16:48:27.719591 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.719490 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp"] Jul 26 16:48:27.720421 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.720318 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43661c45-ce87-45af-a5c7-dd35b0471390" containerName="istio-proxy" Jul 26 16:48:27.720566 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.720424 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="43661c45-ce87-45af-a5c7-dd35b0471390" containerName="istio-proxy" Jul 26 16:48:27.720566 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.720439 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" containerName="istio-proxy" Jul 26 16:48:27.720566 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.720448 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" containerName="istio-proxy" Jul 26 16:48:27.720566 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.720471 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" containerName="istio-proxy" Jul 26 16:48:27.720566 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.720481 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" containerName="istio-proxy" Jul 26 16:48:27.720785 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.720698 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="43661c45-ce87-45af-a5c7-dd35b0471390" containerName="istio-proxy" Jul 26 16:48:27.720785 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.720716 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" containerName="istio-proxy" Jul 26 16:48:27.720785 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.720725 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" containerName="istio-proxy" Jul 26 16:48:27.726651 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.726629 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp" Jul 26 16:48:27.730566 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.730537 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-1b2072\"" Jul 26 16:48:27.730760 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.730542 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-1b2072-dockercfg-8l49q\"" Jul 26 16:48:27.733718 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.733654 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp"] Jul 26 16:48:27.811403 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.811367 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6"] Jul 26 16:48:27.815515 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.815489 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ddd8f8-openshift-default-7ccfff4974-55br6"] Jul 26 16:48:27.827684 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.827658 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjmkh\" (UniqueName: \"kubernetes.io/projected/58f8c546-d3c9-437b-af39-72fb0dfba1de-kube-api-access-sjmkh\") pod \"payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp\" (UID: \"58f8c546-d3c9-437b-af39-72fb0dfba1de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp" Jul 26 16:48:27.827825 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.827801 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f8c546-d3c9-437b-af39-72fb0dfba1de-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp\" (UID: \"58f8c546-d3c9-437b-af39-72fb0dfba1de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp" Jul 26 16:48:27.929242 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.929189 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f8c546-d3c9-437b-af39-72fb0dfba1de-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp\" (UID: \"58f8c546-d3c9-437b-af39-72fb0dfba1de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp" Jul 26 16:48:27.929445 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.929271 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjmkh\" (UniqueName: \"kubernetes.io/projected/58f8c546-d3c9-437b-af39-72fb0dfba1de-kube-api-access-sjmkh\") pod \"payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp\" (UID: \"58f8c546-d3c9-437b-af39-72fb0dfba1de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp" Jul 26 16:48:27.929900 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.929878 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f8c546-d3c9-437b-af39-72fb0dfba1de-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp\" (UID: \"58f8c546-d3c9-437b-af39-72fb0dfba1de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp" Jul 26 16:48:27.938571 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:27.938545 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjmkh\" (UniqueName: \"kubernetes.io/projected/58f8c546-d3c9-437b-af39-72fb0dfba1de-kube-api-access-sjmkh\") pod \"payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp\" (UID: \"58f8c546-d3c9-437b-af39-72fb0dfba1de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp" Jul 26 16:48:28.039511 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:28.039472 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp" Jul 26 16:48:28.190704 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:28.190666 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp"] Jul 26 16:48:28.193317 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:48:28.193286 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f8c546_d3c9_437b_af39_72fb0dfba1de.slice/crio-71dd19030bffabeb6101681bd2eba2971b7e0783b770dddc1af6ffc09cd30e12 WatchSource:0}: Error finding container 71dd19030bffabeb6101681bd2eba2971b7e0783b770dddc1af6ffc09cd30e12: Status 404 returned error can't find the container with id 71dd19030bffabeb6101681bd2eba2971b7e0783b770dddc1af6ffc09cd30e12 Jul 26 16:48:28.487427 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:28.487336 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp" event={"ID":"58f8c546-d3c9-437b-af39-72fb0dfba1de","Type":"ContainerStarted","Data":"71dd19030bffabeb6101681bd2eba2971b7e0783b770dddc1af6ffc09cd30e12"} Jul 26 16:48:28.895271 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:28.895203 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf0342e2-e3d5-4de0-95c2-911ef67ac2b6" path="/var/lib/kubelet/pods/cf0342e2-e3d5-4de0-95c2-911ef67ac2b6/volumes" Jul 26 16:48:30.503129 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:30.503095 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp" event={"ID":"58f8c546-d3c9-437b-af39-72fb0dfba1de","Type":"ContainerStarted","Data":"af8ff4d3784a6a01303c8123235bff47cd691e552221291060fced0e4c655538"} Jul 26 16:48:30.503569 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:30.503238 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp" Jul 26 16:48:30.527158 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:30.527100 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp" podStartSLOduration=1.912488879 podStartE2EDuration="3.527083295s" totalImagesPullingTime="1.614594416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:48:30.525909659 +0000 UTC m=+2384.106996231" watchObservedRunningTime="2026-07-26 16:48:30.527083295 +0000 UTC m=+2384.108169864" Jul 26 16:48:31.521997 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.521964 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m"] Jul 26 16:48:31.525710 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.525677 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.532483 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.532123 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-a06722-openshift-default-dockercfg-dnjtt\"" Jul 26 16:48:31.546820 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.546775 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m"] Jul 26 16:48:31.559665 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.559640 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-data\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.559802 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.559674 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-credential-socket\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.559802 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.559711 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-workload-socket\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.559802 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.559787 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-podinfo\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.559962 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.559826 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istiod-ca-cert\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.559962 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.559904 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-envoy\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.559962 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.559929 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-token\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.560115 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.559963 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlm9x\" (UniqueName: \"kubernetes.io/projected/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-kube-api-access-hlm9x\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.560115 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.560017 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-workload-certs\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.661042 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.661003 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-data\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.661042 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.661046 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-credential-socket\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.661333 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.661070 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-workload-socket\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.661333 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.661092 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-podinfo\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.661333 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.661108 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istiod-ca-cert\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.661333 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.661167 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-envoy\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.661333 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.661193 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-token\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.661333 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.661240 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlm9x\" (UniqueName: \"kubernetes.io/projected/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-kube-api-access-hlm9x\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.661333 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.661267 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-workload-certs\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.661704 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.661594 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-data\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.661704 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.661613 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-workload-socket\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.661704 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.661642 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-credential-socket\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.661704 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.661682 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-workload-certs\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.662359 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.662326 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istiod-ca-cert\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.664930 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.664262 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-envoy\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.664930 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.664467 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-podinfo\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.674009 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.673973 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-token\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.676303 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.676278 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlm9x\" (UniqueName: \"kubernetes.io/projected/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-kube-api-access-hlm9x\") pod \"e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.836604 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.836560 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:31.983979 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.983946 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m"] Jul 26 16:48:31.985664 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:48:31.985634 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0ccea9_8ba4_4d7b_94de_8378bdf67fbb.slice/crio-8e2afbc163801a32f641f13c4a086b0d3311043eea633894cb848f17da7ef6d1 WatchSource:0}: Error finding container 8e2afbc163801a32f641f13c4a086b0d3311043eea633894cb848f17da7ef6d1: Status 404 returned error can't find the container with id 8e2afbc163801a32f641f13c4a086b0d3311043eea633894cb848f17da7ef6d1 Jul 26 16:48:31.990932 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.990897 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:48:31.991034 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.990966 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:48:31.991034 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:31.990995 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:48:32.515937 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:32.515897 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" event={"ID":"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb","Type":"ContainerStarted","Data":"aafed93bb17c5f56f10ccf3ac44ce0bed06263306b49ef0acce4e17ad1fc9521"} Jul 26 16:48:32.516122 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:32.515949 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" event={"ID":"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb","Type":"ContainerStarted","Data":"8e2afbc163801a32f641f13c4a086b0d3311043eea633894cb848f17da7ef6d1"} Jul 26 16:48:32.544954 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:32.544894 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" podStartSLOduration=1.5448791960000001 podStartE2EDuration="1.544879196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:48:32.542328166 +0000 UTC m=+2386.123414736" watchObservedRunningTime="2026-07-26 16:48:32.544879196 +0000 UTC m=+2386.125965765" Jul 26 16:48:32.836829 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:32.836797 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:33.843047 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:33.843012 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:34.527398 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:34.527364 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:34.528407 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:34.528390 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:35.143860 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:35.143825 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj"] Jul 26 16:48:35.147384 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:35.147366 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" Jul 26 16:48:35.150416 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:35.150394 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-a06722-serving-cert\"" Jul 26 16:48:35.163929 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:35.163901 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj"] Jul 26 16:48:35.195281 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:35.195244 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg7mx\" (UniqueName: \"kubernetes.io/projected/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-kube-api-access-tg7mx\") pod \"maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj\" (UID: \"f45c3dcb-dd50-471a-a8d4-b5fc18848a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" Jul 26 16:48:35.195479 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:35.195330 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-maas-api-tls\") pod \"maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj\" (UID: \"f45c3dcb-dd50-471a-a8d4-b5fc18848a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" Jul 26 16:48:35.195479 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:35.195350 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj\" (UID: \"f45c3dcb-dd50-471a-a8d4-b5fc18848a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" Jul 26 16:48:35.296004 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:35.295962 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-maas-api-tls\") pod \"maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj\" (UID: \"f45c3dcb-dd50-471a-a8d4-b5fc18848a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" Jul 26 16:48:35.296004 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:35.296003 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj\" (UID: \"f45c3dcb-dd50-471a-a8d4-b5fc18848a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" Jul 26 16:48:35.296312 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:35.296102 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg7mx\" (UniqueName: \"kubernetes.io/projected/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-kube-api-access-tg7mx\") pod \"maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj\" (UID: \"f45c3dcb-dd50-471a-a8d4-b5fc18848a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" Jul 26 16:48:35.296312 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:48:35.296128 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-serving-cert: secret "maas-api-e2e-shared-b-a06722-serving-cert" not found Jul 26 16:48:35.296312 ip-10-0-130-70 kubenswrapper[2563]: E0726 16:48:35.296207 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-maas-api-tls podName:f45c3dcb-dd50-471a-a8d4-b5fc18848a04 nodeName:}" failed. No retries permitted until 2026-07-26 16:48:35.796182912 +0000 UTC m=+2389.377269479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-maas-api-tls") pod "maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" (UID: "f45c3dcb-dd50-471a-a8d4-b5fc18848a04") : secret "maas-api-e2e-shared-b-a06722-serving-cert" not found Jul 26 16:48:35.296758 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:35.296737 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj\" (UID: \"f45c3dcb-dd50-471a-a8d4-b5fc18848a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" Jul 26 16:48:35.306085 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:35.306061 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg7mx\" (UniqueName: \"kubernetes.io/projected/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-kube-api-access-tg7mx\") pod \"maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj\" (UID: \"f45c3dcb-dd50-471a-a8d4-b5fc18848a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" Jul 26 16:48:35.801882 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:35.801842 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-maas-api-tls\") pod \"maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj\" (UID: \"f45c3dcb-dd50-471a-a8d4-b5fc18848a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" Jul 26 16:48:35.804367 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:35.804337 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-maas-api-tls\") pod \"maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj\" (UID: \"f45c3dcb-dd50-471a-a8d4-b5fc18848a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" Jul 26 16:48:36.065781 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:36.065410 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" Jul 26 16:48:36.254725 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:36.254694 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj"] Jul 26 16:48:36.255896 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:48:36.255869 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45c3dcb_dd50_471a_a8d4_b5fc18848a04.slice/crio-dd31c8308c3e02ace90e9f54835473f42c278ad859d77094737d59513a3d8c18 WatchSource:0}: Error finding container dd31c8308c3e02ace90e9f54835473f42c278ad859d77094737d59513a3d8c18: Status 404 returned error can't find the container with id dd31c8308c3e02ace90e9f54835473f42c278ad859d77094737d59513a3d8c18 Jul 26 16:48:36.540003 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:36.539971 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" event={"ID":"f45c3dcb-dd50-471a-a8d4-b5fc18848a04","Type":"ContainerStarted","Data":"dd31c8308c3e02ace90e9f54835473f42c278ad859d77094737d59513a3d8c18"} Jul 26 16:48:37.547560 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:37.547516 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" event={"ID":"f45c3dcb-dd50-471a-a8d4-b5fc18848a04","Type":"ContainerStarted","Data":"08404b0c7f4c63dd553e19fce5eb6616872a8bb85d2cf4ed93137657d5625eeb"} Jul 26 16:48:37.547936 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:37.547577 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" Jul 26 16:48:37.575443 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:37.575371 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" podStartSLOduration=2.128009705 podStartE2EDuration="2.575347954s" totalImagesPullingTime="447.338249ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:48:37.572994085 +0000 UTC m=+2391.154080657" watchObservedRunningTime="2026-07-26 16:48:37.575347954 +0000 UTC m=+2391.156434526" Jul 26 16:48:41.509386 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:41.509341 2563 patch_prober.go:28] interesting pod/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 26 16:48:41.509869 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:41.509413 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" podUID="a654d489-3ddf-4fd5-b428-f5cba87350c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 26 16:48:41.511485 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:41.511460 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp" Jul 26 16:48:43.559800 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:43.559763 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" Jul 26 16:48:43.587662 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:43.587634 2563 generic.go:356] "Generic (PLEG): container finished" podID="a654d489-3ddf-4fd5-b428-f5cba87350c1" containerID="6aa3ee5275119b02877e894ebcd004a82a8b6a1957fd970d808bbda2174b6c54" exitCode=0 Jul 26 16:48:43.587814 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:43.587709 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" event={"ID":"a654d489-3ddf-4fd5-b428-f5cba87350c1","Type":"ContainerDied","Data":"6aa3ee5275119b02877e894ebcd004a82a8b6a1957fd970d808bbda2174b6c54"} Jul 26 16:48:43.588091 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:43.588075 2563 scope.go:117] "RemoveContainer" containerID="6aa3ee5275119b02877e894ebcd004a82a8b6a1957fd970d808bbda2174b6c54" Jul 26 16:48:43.589291 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:43.589257 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:48:43.589407 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:43.589337 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:48:43.589407 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:43.589370 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:48:44.601018 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:44.600965 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" event={"ID":"a654d489-3ddf-4fd5-b428-f5cba87350c1","Type":"ContainerStarted","Data":"6747b18f0f76caa4729834be1210ea6ae02932bea1534b40bb4bd5f33399ef7d"} Jul 26 16:48:45.404730 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:45.404695 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:45.405115 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:45.405019 2563 patch_prober.go:28] interesting pod/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 26 16:48:45.405340 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:45.405314 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" podUID="a654d489-3ddf-4fd5-b428-f5cba87350c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 26 16:48:46.404620 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:46.404572 2563 patch_prober.go:28] interesting pod/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 26 16:48:46.405141 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:46.404643 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" podUID="a654d489-3ddf-4fd5-b428-f5cba87350c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 26 16:48:47.404537 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:47.404494 2563 patch_prober.go:28] interesting pod/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 26 16:48:47.404809 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:47.404780 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" podUID="a654d489-3ddf-4fd5-b428-f5cba87350c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 26 16:48:48.404301 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:48.404263 2563 patch_prober.go:28] interesting pod/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 26 16:48:48.404496 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:48.404332 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" podUID="a654d489-3ddf-4fd5-b428-f5cba87350c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 26 16:48:49.408162 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:49.408121 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:49.408621 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:49.408553 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:49.409336 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:49.409310 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 16:48:50.569434 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:50.569382 2563 patch_prober.go:28] interesting pod/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 26 16:48:50.569873 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:50.569459 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" podUID="3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 26 16:48:51.652873 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:51.652822 2563 generic.go:356] "Generic (PLEG): container finished" podID="3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" containerID="aafed93bb17c5f56f10ccf3ac44ce0bed06263306b49ef0acce4e17ad1fc9521" exitCode=0 Jul 26 16:48:51.653402 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:51.652902 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" event={"ID":"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb","Type":"ContainerDied","Data":"aafed93bb17c5f56f10ccf3ac44ce0bed06263306b49ef0acce4e17ad1fc9521"} Jul 26 16:48:51.653657 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:51.653640 2563 scope.go:117] "RemoveContainer" containerID="aafed93bb17c5f56f10ccf3ac44ce0bed06263306b49ef0acce4e17ad1fc9521" Jul 26 16:48:51.654680 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:51.654653 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:48:51.654743 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:51.654716 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:48:51.654783 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:51.654746 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:48:52.661714 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:52.661669 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" event={"ID":"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb","Type":"ContainerStarted","Data":"f4b2f0f0060fe43a2b587a719ecec671ef27404ebe2468da7d30dd2bea03da07"} Jul 26 16:48:52.836913 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:52.836880 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:52.837115 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:52.837044 2563 patch_prober.go:28] interesting pod/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 26 16:48:52.837192 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:52.837132 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" podUID="3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 26 16:48:53.837497 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:53.837455 2563 patch_prober.go:28] interesting pod/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 26 16:48:53.837997 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:53.837534 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" podUID="3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 26 16:48:54.837484 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:54.837443 2563 patch_prober.go:28] interesting pod/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 26 16:48:54.837674 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:54.837504 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" podUID="3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 26 16:48:55.837584 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:55.837541 2563 patch_prober.go:28] interesting pod/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 26 16:48:55.837808 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:55.837605 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" podUID="3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 26 16:48:56.840650 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:56.840617 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:56.841065 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:56.840906 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:48:56.841536 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:48:56.841510 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 16:52:47.255426 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:52:47.255398 2563 scope.go:117] "RemoveContainer" containerID="14cb1d3c627495c04ab633bef045ed732e95d428a15a45fce195a5773c61e95c" Jul 26 16:59:16.958768 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:16.958733 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm"] Jul 26 16:59:16.964061 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:16.964030 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:16.967255 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:16.967214 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-50e630-openshift-default-dockercfg-xwb7c\"" Jul 26 16:59:16.974412 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:16.974337 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm"] Jul 26 16:59:17.080380 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.080338 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87hzs\" (UniqueName: \"kubernetes.io/projected/b94162c9-2e19-4b46-ae85-9a420bda7212-kube-api-access-87hzs\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.080556 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.080401 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-workload-certs\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.080556 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.080505 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-data\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.080662 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.080574 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-credential-socket\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.080662 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.080608 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-workload-socket\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.080662 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.080636 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-token\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.080803 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.080702 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94162c9-2e19-4b46-ae85-9a420bda7212-istiod-ca-cert\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.080803 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.080781 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-envoy\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.080896 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.080842 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-podinfo\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.182350 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.182316 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-data\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.182555 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.182372 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-credential-socket\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.182555 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.182496 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-workload-socket\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.182555 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.182535 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-token\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.182714 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.182569 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94162c9-2e19-4b46-ae85-9a420bda7212-istiod-ca-cert\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.182714 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.182626 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-envoy\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.182714 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.182676 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-podinfo\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.182877 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.182717 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87hzs\" (UniqueName: \"kubernetes.io/projected/b94162c9-2e19-4b46-ae85-9a420bda7212-kube-api-access-87hzs\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.182877 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.182783 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-workload-certs\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.182877 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.182805 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-credential-socket\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.182877 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.182862 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-data\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.183273 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.183244 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-workload-socket\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.183447 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.183414 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-workload-certs\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.183699 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.183680 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94162c9-2e19-4b46-ae85-9a420bda7212-istiod-ca-cert\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.185212 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.185193 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-envoy\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.185592 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.185569 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-podinfo\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.192462 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.192441 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-token\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.192570 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.192552 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87hzs\" (UniqueName: \"kubernetes.io/projected/b94162c9-2e19-4b46-ae85-9a420bda7212-kube-api-access-87hzs\") pod \"e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.276915 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.276887 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:17.438794 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.438768 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm"] Jul 26 16:59:17.440807 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:59:17.440782 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94162c9_2e19_4b46_ae85_9a420bda7212.slice/crio-76868400292b400e6e828046e55d24778243adf00eb2a15819ddd46d1411707e WatchSource:0}: Error finding container 76868400292b400e6e828046e55d24778243adf00eb2a15819ddd46d1411707e: Status 404 returned error can't find the container with id 76868400292b400e6e828046e55d24778243adf00eb2a15819ddd46d1411707e Jul 26 16:59:17.443138 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.443105 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:59:17.443249 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.443176 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:59:17.443249 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:17.443205 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:59:18.119257 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:18.119201 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" event={"ID":"b94162c9-2e19-4b46-ae85-9a420bda7212","Type":"ContainerStarted","Data":"47f8ce20fd452cedd4593a3c3e4eb91aee635df0fe4c0d199d83dd739619de74"} Jul 26 16:59:18.119647 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:18.119282 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" event={"ID":"b94162c9-2e19-4b46-ae85-9a420bda7212","Type":"ContainerStarted","Data":"76868400292b400e6e828046e55d24778243adf00eb2a15819ddd46d1411707e"} Jul 26 16:59:18.150280 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:18.150211 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" podStartSLOduration=2.150194887 podStartE2EDuration="2.150194887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:59:18.148129581 +0000 UTC m=+3031.729216155" watchObservedRunningTime="2026-07-26 16:59:18.150194887 +0000 UTC m=+3031.731281457" Jul 26 16:59:18.278082 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:18.278044 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:18.283032 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:18.283004 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:19.125059 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:19.125026 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:19.125972 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:19.125946 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:22.644881 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:22.644848 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf"] Jul 26 16:59:22.648153 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:22.648130 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf" Jul 26 16:59:22.651941 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:22.651913 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-50e630-dockercfg-zpdpg\"" Jul 26 16:59:22.652068 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:22.651961 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-50e630\"" Jul 26 16:59:22.659272 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:22.659237 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf"] Jul 26 16:59:22.742281 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:22.742241 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk76f\" (UniqueName: \"kubernetes.io/projected/9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96-kube-api-access-rk76f\") pod \"payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf\" (UID: \"9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf" Jul 26 16:59:22.742467 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:22.742293 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf\" (UID: \"9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf" Jul 26 16:59:22.843554 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:22.843517 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk76f\" (UniqueName: \"kubernetes.io/projected/9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96-kube-api-access-rk76f\") pod \"payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf\" (UID: \"9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf" Jul 26 16:59:22.843746 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:22.843558 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf\" (UID: \"9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf" Jul 26 16:59:22.844316 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:22.844296 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf\" (UID: \"9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf" Jul 26 16:59:22.855044 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:22.855018 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk76f\" (UniqueName: \"kubernetes.io/projected/9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96-kube-api-access-rk76f\") pod \"payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf\" (UID: \"9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf" Jul 26 16:59:22.961531 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:22.961444 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf" Jul 26 16:59:23.137522 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:23.137490 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf"] Jul 26 16:59:23.139031 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:59:23.139003 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0c2eb8_e9ae_4693_92b8_dccb9c1ada96.slice/crio-42d17e887b69838843161d10c326041aac8da9f9f70f93bdfc2e486712f78b42 WatchSource:0}: Error finding container 42d17e887b69838843161d10c326041aac8da9f9f70f93bdfc2e486712f78b42: Status 404 returned error can't find the container with id 42d17e887b69838843161d10c326041aac8da9f9f70f93bdfc2e486712f78b42 Jul 26 16:59:23.147978 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:23.147950 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf" event={"ID":"9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96","Type":"ContainerStarted","Data":"42d17e887b69838843161d10c326041aac8da9f9f70f93bdfc2e486712f78b42"} Jul 26 16:59:24.154517 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:24.154477 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf" event={"ID":"9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96","Type":"ContainerStarted","Data":"c5dcaa4f6d711bae8daa187ca4487e903af103cd55c156613c3ab09c89ecfcbc"} Jul 26 16:59:24.154930 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:24.154556 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf" Jul 26 16:59:24.178932 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:24.178879 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf" podStartSLOduration=2.1788620659999998 podStartE2EDuration="2.178862066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:59:24.176446091 +0000 UTC m=+3037.757532662" watchObservedRunningTime="2026-07-26 16:59:24.178862066 +0000 UTC m=+3037.759948692" Jul 26 16:59:35.163915 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:35.163869 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf" Jul 26 16:59:38.236153 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:38.236118 2563 generic.go:356] "Generic (PLEG): container finished" podID="b94162c9-2e19-4b46-ae85-9a420bda7212" containerID="47f8ce20fd452cedd4593a3c3e4eb91aee635df0fe4c0d199d83dd739619de74" exitCode=0 Jul 26 16:59:38.236612 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:38.236192 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" event={"ID":"b94162c9-2e19-4b46-ae85-9a420bda7212","Type":"ContainerDied","Data":"47f8ce20fd452cedd4593a3c3e4eb91aee635df0fe4c0d199d83dd739619de74"} Jul 26 16:59:38.236658 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:38.236622 2563 scope.go:117] "RemoveContainer" containerID="47f8ce20fd452cedd4593a3c3e4eb91aee635df0fe4c0d199d83dd739619de74" Jul 26 16:59:38.237937 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:38.237906 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:59:38.238044 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:38.238028 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:59:38.238092 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:38.238067 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:59:39.243004 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:39.242972 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" event={"ID":"b94162c9-2e19-4b46-ae85-9a420bda7212","Type":"ContainerStarted","Data":"7127dbed06e73380450c922765a72e45e90cf72170dbdc536af70258d4df73e0"} Jul 26 16:59:39.277959 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:39.277926 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:39.278335 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:39.278310 2563 patch_prober.go:28] interesting pod/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 26 16:59:39.278503 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:39.278479 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" podUID="b94162c9-2e19-4b46-ae85-9a420bda7212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 26 16:59:40.277668 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:40.277629 2563 patch_prober.go:28] interesting pod/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 26 16:59:40.278302 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:40.278268 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" podUID="b94162c9-2e19-4b46-ae85-9a420bda7212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 26 16:59:41.278008 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:41.277967 2563 patch_prober.go:28] interesting pod/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 26 16:59:41.278627 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:41.278593 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" podUID="b94162c9-2e19-4b46-ae85-9a420bda7212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 26 16:59:42.278023 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:42.277981 2563 patch_prober.go:28] interesting pod/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 26 16:59:42.278545 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:42.278049 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" podUID="b94162c9-2e19-4b46-ae85-9a420bda7212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 26 16:59:43.281678 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:43.281648 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:43.282065 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:43.281900 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:43.282696 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:43.282677 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 16:59:46.957069 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:46.957032 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c"] Jul 26 16:59:46.960906 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:46.960885 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c" Jul 26 16:59:46.965213 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:46.965189 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-930b99\"" Jul 26 16:59:46.965901 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:46.965859 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-930b99-dockercfg-sbmvp\"" Jul 26 16:59:46.970431 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:46.970400 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c"] Jul 26 16:59:46.990831 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:46.990798 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84kbg\" (UniqueName: \"kubernetes.io/projected/643f5bc0-8886-4afb-8f0f-c2d1de1b59a2-kube-api-access-84kbg\") pod \"payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c\" (UID: \"643f5bc0-8886-4afb-8f0f-c2d1de1b59a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c" Jul 26 16:59:46.991101 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:46.991054 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643f5bc0-8886-4afb-8f0f-c2d1de1b59a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c\" (UID: \"643f5bc0-8886-4afb-8f0f-c2d1de1b59a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c" Jul 26 16:59:47.092098 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:47.092061 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643f5bc0-8886-4afb-8f0f-c2d1de1b59a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c\" (UID: \"643f5bc0-8886-4afb-8f0f-c2d1de1b59a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c" Jul 26 16:59:47.092322 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:47.092121 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84kbg\" (UniqueName: \"kubernetes.io/projected/643f5bc0-8886-4afb-8f0f-c2d1de1b59a2-kube-api-access-84kbg\") pod \"payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c\" (UID: \"643f5bc0-8886-4afb-8f0f-c2d1de1b59a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c" Jul 26 16:59:47.092864 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:47.092837 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643f5bc0-8886-4afb-8f0f-c2d1de1b59a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c\" (UID: \"643f5bc0-8886-4afb-8f0f-c2d1de1b59a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c" Jul 26 16:59:47.101724 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:47.101686 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84kbg\" (UniqueName: \"kubernetes.io/projected/643f5bc0-8886-4afb-8f0f-c2d1de1b59a2-kube-api-access-84kbg\") pod \"payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c\" (UID: \"643f5bc0-8886-4afb-8f0f-c2d1de1b59a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c" Jul 26 16:59:47.273446 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:47.273359 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c" Jul 26 16:59:47.415686 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:47.415655 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c"] Jul 26 16:59:47.417129 ip-10-0-130-70 kubenswrapper[2563]: W0726 16:59:47.417104 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643f5bc0_8886_4afb_8f0f_c2d1de1b59a2.slice/crio-89a5fc3d70acbff6fa0f87d76695181ca968c8575355813a7439d985481a552a WatchSource:0}: Error finding container 89a5fc3d70acbff6fa0f87d76695181ca968c8575355813a7439d985481a552a: Status 404 returned error can't find the container with id 89a5fc3d70acbff6fa0f87d76695181ca968c8575355813a7439d985481a552a Jul 26 16:59:48.303649 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:48.303613 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c" event={"ID":"643f5bc0-8886-4afb-8f0f-c2d1de1b59a2","Type":"ContainerStarted","Data":"c501aceaa79a5bf6e1e50ceceddb71d3c7b0fa3bc425f8f376118d4edf7c38e0"} Jul 26 16:59:48.304156 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:48.303665 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c" event={"ID":"643f5bc0-8886-4afb-8f0f-c2d1de1b59a2","Type":"ContainerStarted","Data":"89a5fc3d70acbff6fa0f87d76695181ca968c8575355813a7439d985481a552a"} Jul 26 16:59:48.304156 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:48.303748 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c" Jul 26 16:59:48.322572 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:48.322523 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c" podStartSLOduration=2.322507374 podStartE2EDuration="2.322507374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:59:48.321370737 +0000 UTC m=+3061.902457308" watchObservedRunningTime="2026-07-26 16:59:48.322507374 +0000 UTC m=+3061.903593941" Jul 26 16:59:59.311733 ip-10-0-130-70 kubenswrapper[2563]: I0726 16:59:59.311697 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c" Jul 26 17:00:56.063403 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:56.063365 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf_9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96/payload-processing/0.log" Jul 26 17:00:56.854566 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:56.854531 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp"] Jul 26 17:00:56.858187 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:56.858166 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:56.861108 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:56.861087 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-54f5f8-openshift-default-dockercfg-jkhsk\"" Jul 26 17:00:56.872002 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:56.871972 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp"] Jul 26 17:00:56.978082 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:56.977885 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-credential-socket\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:56.978082 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:56.977938 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab62e80f-a691-4596-9098-1079a14de9ed-istiod-ca-cert\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:56.978082 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:56.977972 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-workload-certs\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:56.978082 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:56.978038 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-istio-envoy\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:56.978488 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:56.978123 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-workload-socket\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:56.978488 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:56.978154 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab62e80f-a691-4596-9098-1079a14de9ed-istio-token\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:56.978488 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:56.978186 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab62e80f-a691-4596-9098-1079a14de9ed-istio-podinfo\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:56.978488 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:56.978216 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhkhv\" (UniqueName: \"kubernetes.io/projected/ab62e80f-a691-4596-9098-1079a14de9ed-kube-api-access-hhkhv\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:56.978488 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:56.978271 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-istio-data\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.079750 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.079713 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-workload-socket\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.079750 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.079752 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab62e80f-a691-4596-9098-1079a14de9ed-istio-token\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.080335 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.079776 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab62e80f-a691-4596-9098-1079a14de9ed-istio-podinfo\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.080335 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.079910 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhkhv\" (UniqueName: \"kubernetes.io/projected/ab62e80f-a691-4596-9098-1079a14de9ed-kube-api-access-hhkhv\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.080335 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.079973 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-istio-data\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.080335 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.080047 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-credential-socket\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.080335 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.080083 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab62e80f-a691-4596-9098-1079a14de9ed-istiod-ca-cert\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.080335 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.080129 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-workload-certs\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.080335 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.080205 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-istio-envoy\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.080335 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.080216 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-workload-socket\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.080335 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.080333 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-istio-data\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.080732 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.080403 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-credential-socket\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.080732 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.080523 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-workload-certs\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.080877 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.080857 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab62e80f-a691-4596-9098-1079a14de9ed-istiod-ca-cert\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.082391 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.082364 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-istio-envoy\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.082625 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.082607 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab62e80f-a691-4596-9098-1079a14de9ed-istio-podinfo\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.092608 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.092584 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab62e80f-a691-4596-9098-1079a14de9ed-istio-token\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.101656 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.101626 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhkhv\" (UniqueName: \"kubernetes.io/projected/ab62e80f-a691-4596-9098-1079a14de9ed-kube-api-access-hhkhv\") pod \"e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.172111 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.172017 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:57.321290 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.321259 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp"] Jul 26 17:00:57.322626 ip-10-0-130-70 kubenswrapper[2563]: W0726 17:00:57.322599 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab62e80f_a691_4596_9098_1079a14de9ed.slice/crio-f614cf7ccbaf1d9979007c89bf4fe695d57316deac3498bcd77169c8fc746cd3 WatchSource:0}: Error finding container f614cf7ccbaf1d9979007c89bf4fe695d57316deac3498bcd77169c8fc746cd3: Status 404 returned error can't find the container with id f614cf7ccbaf1d9979007c89bf4fe695d57316deac3498bcd77169c8fc746cd3 Jul 26 17:00:57.324912 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.324882 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 17:00:57.325001 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.324956 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 17:00:57.325071 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.324999 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 17:00:57.706318 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.706276 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" event={"ID":"ab62e80f-a691-4596-9098-1079a14de9ed","Type":"ContainerStarted","Data":"83f5313d79c6a7ca4534b871c360ff4f077a14676bbf3893ee658ed10680c7ef"} Jul 26 17:00:57.706517 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.706340 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" event={"ID":"ab62e80f-a691-4596-9098-1079a14de9ed","Type":"ContainerStarted","Data":"f614cf7ccbaf1d9979007c89bf4fe695d57316deac3498bcd77169c8fc746cd3"} Jul 26 17:00:57.726686 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:57.726626 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" podStartSLOduration=1.726607402 podStartE2EDuration="1.726607402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:00:57.725397801 +0000 UTC m=+3131.306484382" watchObservedRunningTime="2026-07-26 17:00:57.726607402 +0000 UTC m=+3131.307693971" Jul 26 17:00:58.172303 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:58.172260 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:58.178454 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:58.178417 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:58.712475 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:58.712439 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:00:58.713669 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:00:58.713640 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:01:14.414626 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:14.414235 2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": read tcp 10.134.0.2:39182->10.134.0.71:15021: read: connection reset by peer" start-of-body= Jul 26 17:01:14.414626 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:14.414311 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" podUID="ab62e80f-a691-4596-9098-1079a14de9ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": read tcp 10.134.0.2:39182->10.134.0.71:15021: read: connection reset by peer" Jul 26 17:01:14.824449 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:14.824409 2563 generic.go:356] "Generic (PLEG): container finished" podID="ab62e80f-a691-4596-9098-1079a14de9ed" containerID="83f5313d79c6a7ca4534b871c360ff4f077a14676bbf3893ee658ed10680c7ef" exitCode=0 Jul 26 17:01:14.824640 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:14.824483 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" event={"ID":"ab62e80f-a691-4596-9098-1079a14de9ed","Type":"ContainerDied","Data":"83f5313d79c6a7ca4534b871c360ff4f077a14676bbf3893ee658ed10680c7ef"} Jul 26 17:01:14.824921 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:14.824902 2563 scope.go:117] "RemoveContainer" containerID="83f5313d79c6a7ca4534b871c360ff4f077a14676bbf3893ee658ed10680c7ef" Jul 26 17:01:14.826104 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:14.826079 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 17:01:14.826179 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:14.826164 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 17:01:14.826215 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:14.826193 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 17:01:15.040918 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:15.040882 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l"] Jul 26 17:01:15.043995 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:15.043977 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l" Jul 26 17:01:15.046785 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:15.046765 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2xlcn\"" Jul 26 17:01:15.051827 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:15.051687 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l"] Jul 26 17:01:15.158095 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:15.158002 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322dd3c3-3d4e-46e9-bccc-5e599d3812bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l\" (UID: \"322dd3c3-3d4e-46e9-bccc-5e599d3812bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l" Jul 26 17:01:15.259297 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:15.259248 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322dd3c3-3d4e-46e9-bccc-5e599d3812bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l\" (UID: \"322dd3c3-3d4e-46e9-bccc-5e599d3812bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l" Jul 26 17:01:15.259902 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:15.259878 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322dd3c3-3d4e-46e9-bccc-5e599d3812bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l\" (UID: \"322dd3c3-3d4e-46e9-bccc-5e599d3812bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l" Jul 26 17:01:15.360040 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:15.360004 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l" Jul 26 17:01:15.500733 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:15.500705 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l"] Jul 26 17:01:15.502884 ip-10-0-130-70 kubenswrapper[2563]: W0726 17:01:15.502856 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322dd3c3_3d4e_46e9_bccc_5e599d3812bb.slice/crio-738ef4a433487075d0e6e39eba6104c3133ed6de82e83c8f96cd1130893fc7bf WatchSource:0}: Error finding container 738ef4a433487075d0e6e39eba6104c3133ed6de82e83c8f96cd1130893fc7bf: Status 404 returned error can't find the container with id 738ef4a433487075d0e6e39eba6104c3133ed6de82e83c8f96cd1130893fc7bf Jul 26 17:01:15.832269 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:15.832210 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" event={"ID":"ab62e80f-a691-4596-9098-1079a14de9ed","Type":"ContainerStarted","Data":"a80dbd28793153d4b4537a74124b81703bb89ae208e491d40e56e2afe4dbbf84"} Jul 26 17:01:15.833773 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:15.833734 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l" event={"ID":"322dd3c3-3d4e-46e9-bccc-5e599d3812bb","Type":"ContainerStarted","Data":"4288ef852f8eef78e85c5d94753befac73ad756a6eb9da6c258d559dbd87ee16"} Jul 26 17:01:15.833897 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:15.833789 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l" event={"ID":"322dd3c3-3d4e-46e9-bccc-5e599d3812bb","Type":"ContainerStarted","Data":"738ef4a433487075d0e6e39eba6104c3133ed6de82e83c8f96cd1130893fc7bf"} Jul 26 17:01:15.868777 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:15.868723 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l" podStartSLOduration=0.868707526 podStartE2EDuration="868.707526ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:01:15.865422859 +0000 UTC m=+3149.446509432" watchObservedRunningTime="2026-07-26 17:01:15.868707526 +0000 UTC m=+3149.449794117" Jul 26 17:01:16.172493 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:16.172452 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:01:16.172733 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:16.172705 2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 26 17:01:16.172809 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:16.172761 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" podUID="ab62e80f-a691-4596-9098-1079a14de9ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 26 17:01:16.840784 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:16.840740 2563 generic.go:356] "Generic (PLEG): container finished" podID="322dd3c3-3d4e-46e9-bccc-5e599d3812bb" containerID="4288ef852f8eef78e85c5d94753befac73ad756a6eb9da6c258d559dbd87ee16" exitCode=0 Jul 26 17:01:16.841256 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:16.840828 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l" event={"ID":"322dd3c3-3d4e-46e9-bccc-5e599d3812bb","Type":"ContainerDied","Data":"4288ef852f8eef78e85c5d94753befac73ad756a6eb9da6c258d559dbd87ee16"} Jul 26 17:01:17.173119 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:17.173019 2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 26 17:01:17.173422 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:17.173390 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" podUID="ab62e80f-a691-4596-9098-1079a14de9ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 26 17:01:18.020151 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:18.020127 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l" Jul 26 17:01:18.092491 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:18.092458 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322dd3c3-3d4e-46e9-bccc-5e599d3812bb-maas-api-service-ca\") pod \"322dd3c3-3d4e-46e9-bccc-5e599d3812bb\" (UID: \"322dd3c3-3d4e-46e9-bccc-5e599d3812bb\") " Jul 26 17:01:18.092928 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:18.092899 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322dd3c3-3d4e-46e9-bccc-5e599d3812bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "322dd3c3-3d4e-46e9-bccc-5e599d3812bb" (UID: "322dd3c3-3d4e-46e9-bccc-5e599d3812bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 17:01:18.173305 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:18.172699 2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 26 17:01:18.173305 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:18.172767 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" podUID="ab62e80f-a691-4596-9098-1079a14de9ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 26 17:01:18.193313 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:18.193285 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322dd3c3-3d4e-46e9-bccc-5e599d3812bb-maas-api-service-ca\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:01:18.853850 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:18.853817 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l" Jul 26 17:01:18.854032 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:18.853860 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l" event={"ID":"322dd3c3-3d4e-46e9-bccc-5e599d3812bb","Type":"ContainerDied","Data":"738ef4a433487075d0e6e39eba6104c3133ed6de82e83c8f96cd1130893fc7bf"} Jul 26 17:01:18.854032 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:18.853914 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="738ef4a433487075d0e6e39eba6104c3133ed6de82e83c8f96cd1130893fc7bf" Jul 26 17:01:19.172965 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:19.172847 2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 26 17:01:19.173578 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:19.173521 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" podUID="ab62e80f-a691-4596-9098-1079a14de9ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 26 17:01:20.176054 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:20.176021 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:01:20.176494 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:20.176348 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:01:20.176987 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:20.176970 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:01:30.119836 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:30.119794 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l"] Jul 26 17:01:30.126281 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:30.126251 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-54f5f8-z9w6l"] Jul 26 17:01:30.891535 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:30.891499 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="322dd3c3-3d4e-46e9-bccc-5e599d3812bb" path="/var/lib/kubelet/pods/322dd3c3-3d4e-46e9-bccc-5e599d3812bb/volumes" Jul 26 17:01:37.123693 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:37.123653 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp"] Jul 26 17:01:37.124091 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:37.123947 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" podUID="ab62e80f-a691-4596-9098-1079a14de9ed" containerName="istio-proxy" containerID="cri-o://a80dbd28793153d4b4537a74124b81703bb89ae208e491d40e56e2afe4dbbf84" gracePeriod=30 Jul 26 17:01:42.393517 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.393450 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:01:42.546010 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.545980 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-workload-socket\") pod \"ab62e80f-a691-4596-9098-1079a14de9ed\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " Jul 26 17:01:42.546010 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.546015 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-istio-data\") pod \"ab62e80f-a691-4596-9098-1079a14de9ed\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " Jul 26 17:01:42.546284 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.546058 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-istio-envoy\") pod \"ab62e80f-a691-4596-9098-1079a14de9ed\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " Jul 26 17:01:42.546284 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.546072 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-credential-socket\") pod \"ab62e80f-a691-4596-9098-1079a14de9ed\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " Jul 26 17:01:42.546284 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.546104 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-workload-certs\") pod \"ab62e80f-a691-4596-9098-1079a14de9ed\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " Jul 26 17:01:42.546284 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.546173 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab62e80f-a691-4596-9098-1079a14de9ed-istio-token\") pod \"ab62e80f-a691-4596-9098-1079a14de9ed\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " Jul 26 17:01:42.546284 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.546197 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhkhv\" (UniqueName: \"kubernetes.io/projected/ab62e80f-a691-4596-9098-1079a14de9ed-kube-api-access-hhkhv\") pod \"ab62e80f-a691-4596-9098-1079a14de9ed\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " Jul 26 17:01:42.546284 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.546252 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab62e80f-a691-4596-9098-1079a14de9ed-istio-podinfo\") pod \"ab62e80f-a691-4596-9098-1079a14de9ed\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " Jul 26 17:01:42.546618 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.546297 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab62e80f-a691-4596-9098-1079a14de9ed-istiod-ca-cert\") pod \"ab62e80f-a691-4596-9098-1079a14de9ed\" (UID: \"ab62e80f-a691-4596-9098-1079a14de9ed\") " Jul 26 17:01:42.546618 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.546311 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ab62e80f-a691-4596-9098-1079a14de9ed" (UID: "ab62e80f-a691-4596-9098-1079a14de9ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:01:42.546618 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.546470 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ab62e80f-a691-4596-9098-1079a14de9ed" (UID: "ab62e80f-a691-4596-9098-1079a14de9ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:01:42.546837 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.546639 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-workload-certs\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:01:42.546837 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.546659 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-workload-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:01:42.546837 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.546680 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ab62e80f-a691-4596-9098-1079a14de9ed" (UID: "ab62e80f-a691-4596-9098-1079a14de9ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:01:42.546837 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.546740 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab62e80f-a691-4596-9098-1079a14de9ed-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ab62e80f-a691-4596-9098-1079a14de9ed" (UID: "ab62e80f-a691-4596-9098-1079a14de9ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 17:01:42.547056 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.547018 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-istio-data" (OuterVolumeSpecName: "istio-data") pod "ab62e80f-a691-4596-9098-1079a14de9ed" (UID: "ab62e80f-a691-4596-9098-1079a14de9ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:01:42.548264 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.548236 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ab62e80f-a691-4596-9098-1079a14de9ed" (UID: "ab62e80f-a691-4596-9098-1079a14de9ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:01:42.548404 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.548384 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab62e80f-a691-4596-9098-1079a14de9ed-istio-token" (OuterVolumeSpecName: "istio-token") pod "ab62e80f-a691-4596-9098-1079a14de9ed" (UID: "ab62e80f-a691-4596-9098-1079a14de9ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:01:42.548475 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.548424 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab62e80f-a691-4596-9098-1079a14de9ed-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ab62e80f-a691-4596-9098-1079a14de9ed" (UID: "ab62e80f-a691-4596-9098-1079a14de9ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 17:01:42.548475 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.548459 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab62e80f-a691-4596-9098-1079a14de9ed-kube-api-access-hhkhv" (OuterVolumeSpecName: "kube-api-access-hhkhv") pod "ab62e80f-a691-4596-9098-1079a14de9ed" (UID: "ab62e80f-a691-4596-9098-1079a14de9ed"). InnerVolumeSpecName "kube-api-access-hhkhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:01:42.647750 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.647662 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab62e80f-a691-4596-9098-1079a14de9ed-istiod-ca-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:01:42.647750 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.647690 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-istio-data\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:01:42.647750 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.647702 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-istio-envoy\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:01:42.647750 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.647712 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab62e80f-a691-4596-9098-1079a14de9ed-credential-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:01:42.647750 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.647720 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab62e80f-a691-4596-9098-1079a14de9ed-istio-token\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:01:42.647750 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.647729 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhkhv\" (UniqueName: \"kubernetes.io/projected/ab62e80f-a691-4596-9098-1079a14de9ed-kube-api-access-hhkhv\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:01:42.647750 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:42.647737 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab62e80f-a691-4596-9098-1079a14de9ed-istio-podinfo\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:01:43.000732 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:43.000695 2563 generic.go:356] "Generic (PLEG): container finished" podID="ab62e80f-a691-4596-9098-1079a14de9ed" containerID="a80dbd28793153d4b4537a74124b81703bb89ae208e491d40e56e2afe4dbbf84" exitCode=0 Jul 26 17:01:43.000944 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:43.000745 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" event={"ID":"ab62e80f-a691-4596-9098-1079a14de9ed","Type":"ContainerDied","Data":"a80dbd28793153d4b4537a74124b81703bb89ae208e491d40e56e2afe4dbbf84"} Jul 26 17:01:43.000944 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:43.000754 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" Jul 26 17:01:43.000944 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:43.000791 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp" event={"ID":"ab62e80f-a691-4596-9098-1079a14de9ed","Type":"ContainerDied","Data":"f614cf7ccbaf1d9979007c89bf4fe695d57316deac3498bcd77169c8fc746cd3"} Jul 26 17:01:43.000944 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:43.000813 2563 scope.go:117] "RemoveContainer" containerID="a80dbd28793153d4b4537a74124b81703bb89ae208e491d40e56e2afe4dbbf84" Jul 26 17:01:43.011788 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:43.011761 2563 scope.go:117] "RemoveContainer" containerID="83f5313d79c6a7ca4534b871c360ff4f077a14676bbf3893ee658ed10680c7ef" Jul 26 17:01:43.021172 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:43.021151 2563 scope.go:117] "RemoveContainer" containerID="a80dbd28793153d4b4537a74124b81703bb89ae208e491d40e56e2afe4dbbf84" Jul 26 17:01:43.021471 ip-10-0-130-70 kubenswrapper[2563]: E0726 17:01:43.021450 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80dbd28793153d4b4537a74124b81703bb89ae208e491d40e56e2afe4dbbf84\": container with ID starting with a80dbd28793153d4b4537a74124b81703bb89ae208e491d40e56e2afe4dbbf84 not found: ID does not exist" containerID="a80dbd28793153d4b4537a74124b81703bb89ae208e491d40e56e2afe4dbbf84" Jul 26 17:01:43.021553 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:43.021482 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80dbd28793153d4b4537a74124b81703bb89ae208e491d40e56e2afe4dbbf84"} err="failed to get container status \"a80dbd28793153d4b4537a74124b81703bb89ae208e491d40e56e2afe4dbbf84\": rpc error: code = NotFound desc = could not find container \"a80dbd28793153d4b4537a74124b81703bb89ae208e491d40e56e2afe4dbbf84\": container with ID starting with a80dbd28793153d4b4537a74124b81703bb89ae208e491d40e56e2afe4dbbf84 not found: ID does not exist" Jul 26 17:01:43.021553 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:43.021503 2563 scope.go:117] "RemoveContainer" containerID="83f5313d79c6a7ca4534b871c360ff4f077a14676bbf3893ee658ed10680c7ef" Jul 26 17:01:43.021775 ip-10-0-130-70 kubenswrapper[2563]: E0726 17:01:43.021754 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f5313d79c6a7ca4534b871c360ff4f077a14676bbf3893ee658ed10680c7ef\": container with ID starting with 83f5313d79c6a7ca4534b871c360ff4f077a14676bbf3893ee658ed10680c7ef not found: ID does not exist" containerID="83f5313d79c6a7ca4534b871c360ff4f077a14676bbf3893ee658ed10680c7ef" Jul 26 17:01:43.021822 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:43.021781 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f5313d79c6a7ca4534b871c360ff4f077a14676bbf3893ee658ed10680c7ef"} err="failed to get container status \"83f5313d79c6a7ca4534b871c360ff4f077a14676bbf3893ee658ed10680c7ef\": rpc error: code = NotFound desc = could not find container \"83f5313d79c6a7ca4534b871c360ff4f077a14676bbf3893ee658ed10680c7ef\": container with ID starting with 83f5313d79c6a7ca4534b871c360ff4f077a14676bbf3893ee658ed10680c7ef not found: ID does not exist" Jul 26 17:01:43.030694 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:43.030650 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp"] Jul 26 17:01:43.034016 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:43.033993 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-54f5f8-openshift-default-89d78d84-w9gvp"] Jul 26 17:01:44.890642 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:01:44.890606 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab62e80f-a691-4596-9098-1079a14de9ed" path="/var/lib/kubelet/pods/ab62e80f-a691-4596-9098-1079a14de9ed/volumes" Jul 26 17:02:01.225855 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:01.225821 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf"] Jul 26 17:02:01.226398 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:01.226050 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf" podUID="9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96" containerName="payload-processing" containerID="cri-o://c5dcaa4f6d711bae8daa187ca4487e903af103cd55c156613c3ab09c89ecfcbc" gracePeriod=30 Jul 26 17:02:01.518469 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:01.518438 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf" Jul 26 17:02:01.618026 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:01.617988 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk76f\" (UniqueName: \"kubernetes.io/projected/9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96-kube-api-access-rk76f\") pod \"9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96\" (UID: \"9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96\") " Jul 26 17:02:01.618257 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:01.618160 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96-plugins-config-volume\") pod \"9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96\" (UID: \"9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96\") " Jul 26 17:02:01.618542 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:01.618519 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96" (UID: "9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 17:02:01.620057 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:01.620034 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96-kube-api-access-rk76f" (OuterVolumeSpecName: "kube-api-access-rk76f") pod "9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96" (UID: "9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96"). InnerVolumeSpecName "kube-api-access-rk76f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:02:01.719312 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:01.719277 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96-plugins-config-volume\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:02:01.719312 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:01.719307 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk76f\" (UniqueName: \"kubernetes.io/projected/9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96-kube-api-access-rk76f\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:02:02.114898 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:02.114859 2563 generic.go:356] "Generic (PLEG): container finished" podID="9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96" containerID="c5dcaa4f6d711bae8daa187ca4487e903af103cd55c156613c3ab09c89ecfcbc" exitCode=0 Jul 26 17:02:02.115083 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:02.114902 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf" Jul 26 17:02:02.115083 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:02.114945 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf" event={"ID":"9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96","Type":"ContainerDied","Data":"c5dcaa4f6d711bae8daa187ca4487e903af103cd55c156613c3ab09c89ecfcbc"} Jul 26 17:02:02.115083 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:02.114997 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf" event={"ID":"9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96","Type":"ContainerDied","Data":"42d17e887b69838843161d10c326041aac8da9f9f70f93bdfc2e486712f78b42"} Jul 26 17:02:02.115083 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:02.115022 2563 scope.go:117] "RemoveContainer" containerID="c5dcaa4f6d711bae8daa187ca4487e903af103cd55c156613c3ab09c89ecfcbc" Jul 26 17:02:02.124779 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:02.124758 2563 scope.go:117] "RemoveContainer" containerID="c5dcaa4f6d711bae8daa187ca4487e903af103cd55c156613c3ab09c89ecfcbc" Jul 26 17:02:02.125047 ip-10-0-130-70 kubenswrapper[2563]: E0726 17:02:02.125028 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5dcaa4f6d711bae8daa187ca4487e903af103cd55c156613c3ab09c89ecfcbc\": container with ID starting with c5dcaa4f6d711bae8daa187ca4487e903af103cd55c156613c3ab09c89ecfcbc not found: ID does not exist" containerID="c5dcaa4f6d711bae8daa187ca4487e903af103cd55c156613c3ab09c89ecfcbc" Jul 26 17:02:02.125106 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:02.125054 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5dcaa4f6d711bae8daa187ca4487e903af103cd55c156613c3ab09c89ecfcbc"} err="failed to get container status \"c5dcaa4f6d711bae8daa187ca4487e903af103cd55c156613c3ab09c89ecfcbc\": rpc error: code = NotFound desc = could not find container \"c5dcaa4f6d711bae8daa187ca4487e903af103cd55c156613c3ab09c89ecfcbc\": container with ID starting with c5dcaa4f6d711bae8daa187ca4487e903af103cd55c156613c3ab09c89ecfcbc not found: ID does not exist" Jul 26 17:02:02.138115 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:02.138089 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf"] Jul 26 17:02:02.146767 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:02.146745 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-50e630-75b6cdc5fb-klkhf"] Jul 26 17:02:02.896240 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:02:02.896180 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96" path="/var/lib/kubelet/pods/9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96/volumes" Jul 26 17:03:09.844463 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:09.844419 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm"] Jul 26 17:03:09.844966 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:09.844695 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" podUID="b94162c9-2e19-4b46-ae85-9a420bda7212" containerName="istio-proxy" containerID="cri-o://7127dbed06e73380450c922765a72e45e90cf72170dbdc536af70258d4df73e0" gracePeriod=30 Jul 26 17:03:13.284308 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:13.284264 2563 patch_prober.go:28] interesting pod/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 17:03:13.284756 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:13.284342 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" podUID="b94162c9-2e19-4b46-ae85-9a420bda7212" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 17:03:15.114662 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.114596 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 17:03:15.223973 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.223930 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-token\") pod \"b94162c9-2e19-4b46-ae85-9a420bda7212\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " Jul 26 17:03:15.224176 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.224074 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-podinfo\") pod \"b94162c9-2e19-4b46-ae85-9a420bda7212\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " Jul 26 17:03:15.224176 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.224115 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87hzs\" (UniqueName: \"kubernetes.io/projected/b94162c9-2e19-4b46-ae85-9a420bda7212-kube-api-access-87hzs\") pod \"b94162c9-2e19-4b46-ae85-9a420bda7212\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " Jul 26 17:03:15.224319 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.224177 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-envoy\") pod \"b94162c9-2e19-4b46-ae85-9a420bda7212\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " Jul 26 17:03:15.224319 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.224205 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-credential-socket\") pod \"b94162c9-2e19-4b46-ae85-9a420bda7212\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " Jul 26 17:03:15.224319 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.224247 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-workload-socket\") pod \"b94162c9-2e19-4b46-ae85-9a420bda7212\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " Jul 26 17:03:15.224319 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.224316 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94162c9-2e19-4b46-ae85-9a420bda7212-istiod-ca-cert\") pod \"b94162c9-2e19-4b46-ae85-9a420bda7212\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " Jul 26 17:03:15.224540 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.224351 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-workload-certs\") pod \"b94162c9-2e19-4b46-ae85-9a420bda7212\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " Jul 26 17:03:15.224540 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.224385 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-data\") pod \"b94162c9-2e19-4b46-ae85-9a420bda7212\" (UID: \"b94162c9-2e19-4b46-ae85-9a420bda7212\") " Jul 26 17:03:15.224649 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.224546 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b94162c9-2e19-4b46-ae85-9a420bda7212" (UID: "b94162c9-2e19-4b46-ae85-9a420bda7212"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:03:15.224649 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.224585 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b94162c9-2e19-4b46-ae85-9a420bda7212" (UID: "b94162c9-2e19-4b46-ae85-9a420bda7212"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:03:15.224850 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.224809 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b94162c9-2e19-4b46-ae85-9a420bda7212" (UID: "b94162c9-2e19-4b46-ae85-9a420bda7212"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:03:15.224977 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.224883 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-credential-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:15.224977 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.224904 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-workload-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:15.225094 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.225067 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94162c9-2e19-4b46-ae85-9a420bda7212-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b94162c9-2e19-4b46-ae85-9a420bda7212" (UID: "b94162c9-2e19-4b46-ae85-9a420bda7212"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 17:03:15.225584 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.225551 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-data" (OuterVolumeSpecName: "istio-data") pod "b94162c9-2e19-4b46-ae85-9a420bda7212" (UID: "b94162c9-2e19-4b46-ae85-9a420bda7212"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:03:15.226346 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.226324 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b94162c9-2e19-4b46-ae85-9a420bda7212" (UID: "b94162c9-2e19-4b46-ae85-9a420bda7212"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:03:15.226603 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.226587 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b94162c9-2e19-4b46-ae85-9a420bda7212" (UID: "b94162c9-2e19-4b46-ae85-9a420bda7212"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 17:03:15.226669 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.226620 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-token" (OuterVolumeSpecName: "istio-token") pod "b94162c9-2e19-4b46-ae85-9a420bda7212" (UID: "b94162c9-2e19-4b46-ae85-9a420bda7212"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:03:15.226719 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.226685 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94162c9-2e19-4b46-ae85-9a420bda7212-kube-api-access-87hzs" (OuterVolumeSpecName: "kube-api-access-87hzs") pod "b94162c9-2e19-4b46-ae85-9a420bda7212" (UID: "b94162c9-2e19-4b46-ae85-9a420bda7212"). InnerVolumeSpecName "kube-api-access-87hzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:03:15.326478 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.326441 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94162c9-2e19-4b46-ae85-9a420bda7212-istiod-ca-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:15.326478 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.326472 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-workload-certs\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:15.326478 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.326481 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-data\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:15.326710 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.326489 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-token\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:15.326710 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.326496 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-podinfo\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:15.326710 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.326504 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87hzs\" (UniqueName: \"kubernetes.io/projected/b94162c9-2e19-4b46-ae85-9a420bda7212-kube-api-access-87hzs\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:15.326710 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.326513 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94162c9-2e19-4b46-ae85-9a420bda7212-istio-envoy\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:15.533359 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.533322 2563 generic.go:356] "Generic (PLEG): container finished" podID="b94162c9-2e19-4b46-ae85-9a420bda7212" containerID="7127dbed06e73380450c922765a72e45e90cf72170dbdc536af70258d4df73e0" exitCode=0 Jul 26 17:03:15.533553 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.533368 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" event={"ID":"b94162c9-2e19-4b46-ae85-9a420bda7212","Type":"ContainerDied","Data":"7127dbed06e73380450c922765a72e45e90cf72170dbdc536af70258d4df73e0"} Jul 26 17:03:15.533553 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.533386 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" Jul 26 17:03:15.533553 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.533447 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm" event={"ID":"b94162c9-2e19-4b46-ae85-9a420bda7212","Type":"ContainerDied","Data":"76868400292b400e6e828046e55d24778243adf00eb2a15819ddd46d1411707e"} Jul 26 17:03:15.533553 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.533473 2563 scope.go:117] "RemoveContainer" containerID="7127dbed06e73380450c922765a72e45e90cf72170dbdc536af70258d4df73e0" Jul 26 17:03:15.543750 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.543730 2563 scope.go:117] "RemoveContainer" containerID="47f8ce20fd452cedd4593a3c3e4eb91aee635df0fe4c0d199d83dd739619de74" Jul 26 17:03:15.552344 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.552324 2563 scope.go:117] "RemoveContainer" containerID="7127dbed06e73380450c922765a72e45e90cf72170dbdc536af70258d4df73e0" Jul 26 17:03:15.552587 ip-10-0-130-70 kubenswrapper[2563]: E0726 17:03:15.552567 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7127dbed06e73380450c922765a72e45e90cf72170dbdc536af70258d4df73e0\": container with ID starting with 7127dbed06e73380450c922765a72e45e90cf72170dbdc536af70258d4df73e0 not found: ID does not exist" containerID="7127dbed06e73380450c922765a72e45e90cf72170dbdc536af70258d4df73e0" Jul 26 17:03:15.552633 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.552597 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7127dbed06e73380450c922765a72e45e90cf72170dbdc536af70258d4df73e0"} err="failed to get container status \"7127dbed06e73380450c922765a72e45e90cf72170dbdc536af70258d4df73e0\": rpc error: code = NotFound desc = could not find container \"7127dbed06e73380450c922765a72e45e90cf72170dbdc536af70258d4df73e0\": container with ID starting with 7127dbed06e73380450c922765a72e45e90cf72170dbdc536af70258d4df73e0 not found: ID does not exist" Jul 26 17:03:15.552633 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.552614 2563 scope.go:117] "RemoveContainer" containerID="47f8ce20fd452cedd4593a3c3e4eb91aee635df0fe4c0d199d83dd739619de74" Jul 26 17:03:15.552872 ip-10-0-130-70 kubenswrapper[2563]: E0726 17:03:15.552853 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f8ce20fd452cedd4593a3c3e4eb91aee635df0fe4c0d199d83dd739619de74\": container with ID starting with 47f8ce20fd452cedd4593a3c3e4eb91aee635df0fe4c0d199d83dd739619de74 not found: ID does not exist" containerID="47f8ce20fd452cedd4593a3c3e4eb91aee635df0fe4c0d199d83dd739619de74" Jul 26 17:03:15.552925 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.552878 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f8ce20fd452cedd4593a3c3e4eb91aee635df0fe4c0d199d83dd739619de74"} err="failed to get container status \"47f8ce20fd452cedd4593a3c3e4eb91aee635df0fe4c0d199d83dd739619de74\": rpc error: code = NotFound desc = could not find container \"47f8ce20fd452cedd4593a3c3e4eb91aee635df0fe4c0d199d83dd739619de74\": container with ID starting with 47f8ce20fd452cedd4593a3c3e4eb91aee635df0fe4c0d199d83dd739619de74 not found: ID does not exist" Jul 26 17:03:15.556803 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.556778 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm"] Jul 26 17:03:15.560337 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:15.560315 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-50e630-openshift-default-9796ff68-g5dtm"] Jul 26 17:03:16.891413 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:16.891378 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b94162c9-2e19-4b46-ae85-9a420bda7212" path="/var/lib/kubelet/pods/b94162c9-2e19-4b46-ae85-9a420bda7212/volumes" Jul 26 17:03:21.588970 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:21.588932 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c"] Jul 26 17:03:21.589429 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:21.589182 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c" podUID="643f5bc0-8886-4afb-8f0f-c2d1de1b59a2" containerName="payload-pre-processing" containerID="cri-o://c501aceaa79a5bf6e1e50ceceddb71d3c7b0fa3bc425f8f376118d4edf7c38e0" gracePeriod=30 Jul 26 17:03:21.855333 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:21.855307 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c" Jul 26 17:03:21.885626 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:21.885596 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84kbg\" (UniqueName: \"kubernetes.io/projected/643f5bc0-8886-4afb-8f0f-c2d1de1b59a2-kube-api-access-84kbg\") pod \"643f5bc0-8886-4afb-8f0f-c2d1de1b59a2\" (UID: \"643f5bc0-8886-4afb-8f0f-c2d1de1b59a2\") " Jul 26 17:03:21.885802 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:21.885635 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643f5bc0-8886-4afb-8f0f-c2d1de1b59a2-plugins-config-volume\") pod \"643f5bc0-8886-4afb-8f0f-c2d1de1b59a2\" (UID: \"643f5bc0-8886-4afb-8f0f-c2d1de1b59a2\") " Jul 26 17:03:21.886058 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:21.886021 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643f5bc0-8886-4afb-8f0f-c2d1de1b59a2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "643f5bc0-8886-4afb-8f0f-c2d1de1b59a2" (UID: "643f5bc0-8886-4afb-8f0f-c2d1de1b59a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 17:03:21.887759 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:21.887736 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643f5bc0-8886-4afb-8f0f-c2d1de1b59a2-kube-api-access-84kbg" (OuterVolumeSpecName: "kube-api-access-84kbg") pod "643f5bc0-8886-4afb-8f0f-c2d1de1b59a2" (UID: "643f5bc0-8886-4afb-8f0f-c2d1de1b59a2"). InnerVolumeSpecName "kube-api-access-84kbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:03:21.986401 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:21.986355 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84kbg\" (UniqueName: \"kubernetes.io/projected/643f5bc0-8886-4afb-8f0f-c2d1de1b59a2-kube-api-access-84kbg\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:21.986401 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:21.986395 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643f5bc0-8886-4afb-8f0f-c2d1de1b59a2-plugins-config-volume\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:22.575675 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:22.575639 2563 generic.go:356] "Generic (PLEG): container finished" podID="643f5bc0-8886-4afb-8f0f-c2d1de1b59a2" containerID="c501aceaa79a5bf6e1e50ceceddb71d3c7b0fa3bc425f8f376118d4edf7c38e0" exitCode=0 Jul 26 17:03:22.575847 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:22.575690 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c" Jul 26 17:03:22.575847 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:22.575694 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c" event={"ID":"643f5bc0-8886-4afb-8f0f-c2d1de1b59a2","Type":"ContainerDied","Data":"c501aceaa79a5bf6e1e50ceceddb71d3c7b0fa3bc425f8f376118d4edf7c38e0"} Jul 26 17:03:22.575847 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:22.575750 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c" event={"ID":"643f5bc0-8886-4afb-8f0f-c2d1de1b59a2","Type":"ContainerDied","Data":"89a5fc3d70acbff6fa0f87d76695181ca968c8575355813a7439d985481a552a"} Jul 26 17:03:22.575847 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:22.575780 2563 scope.go:117] "RemoveContainer" containerID="c501aceaa79a5bf6e1e50ceceddb71d3c7b0fa3bc425f8f376118d4edf7c38e0" Jul 26 17:03:22.587104 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:22.586623 2563 scope.go:117] "RemoveContainer" containerID="c501aceaa79a5bf6e1e50ceceddb71d3c7b0fa3bc425f8f376118d4edf7c38e0" Jul 26 17:03:22.588144 ip-10-0-130-70 kubenswrapper[2563]: E0726 17:03:22.588108 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c501aceaa79a5bf6e1e50ceceddb71d3c7b0fa3bc425f8f376118d4edf7c38e0\": container with ID starting with c501aceaa79a5bf6e1e50ceceddb71d3c7b0fa3bc425f8f376118d4edf7c38e0 not found: ID does not exist" containerID="c501aceaa79a5bf6e1e50ceceddb71d3c7b0fa3bc425f8f376118d4edf7c38e0" Jul 26 17:03:22.588301 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:22.588152 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c501aceaa79a5bf6e1e50ceceddb71d3c7b0fa3bc425f8f376118d4edf7c38e0"} err="failed to get container status \"c501aceaa79a5bf6e1e50ceceddb71d3c7b0fa3bc425f8f376118d4edf7c38e0\": rpc error: code = NotFound desc = could not find container \"c501aceaa79a5bf6e1e50ceceddb71d3c7b0fa3bc425f8f376118d4edf7c38e0\": container with ID starting with c501aceaa79a5bf6e1e50ceceddb71d3c7b0fa3bc425f8f376118d4edf7c38e0 not found: ID does not exist" Jul 26 17:03:22.601396 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:22.601372 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c"] Jul 26 17:03:22.605400 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:22.605380 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-930b99-8664569964-2nh6c"] Jul 26 17:03:22.892207 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:22.892127 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="643f5bc0-8886-4afb-8f0f-c2d1de1b59a2" path="/var/lib/kubelet/pods/643f5bc0-8886-4afb-8f0f-c2d1de1b59a2/volumes" Jul 26 17:03:43.989641 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:43.989605 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp"] Jul 26 17:03:43.990134 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:43.989811 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp" podUID="58f8c546-d3c9-437b-af39-72fb0dfba1de" containerName="payload-processing" containerID="cri-o://af8ff4d3784a6a01303c8123235bff47cd691e552221291060fced0e4c655538" gracePeriod=30 Jul 26 17:03:44.249503 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:44.249441 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp" Jul 26 17:03:44.405309 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:44.405272 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjmkh\" (UniqueName: \"kubernetes.io/projected/58f8c546-d3c9-437b-af39-72fb0dfba1de-kube-api-access-sjmkh\") pod \"58f8c546-d3c9-437b-af39-72fb0dfba1de\" (UID: \"58f8c546-d3c9-437b-af39-72fb0dfba1de\") " Jul 26 17:03:44.405513 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:44.405316 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f8c546-d3c9-437b-af39-72fb0dfba1de-plugins-config-volume\") pod \"58f8c546-d3c9-437b-af39-72fb0dfba1de\" (UID: \"58f8c546-d3c9-437b-af39-72fb0dfba1de\") " Jul 26 17:03:44.405751 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:44.405713 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f8c546-d3c9-437b-af39-72fb0dfba1de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "58f8c546-d3c9-437b-af39-72fb0dfba1de" (UID: "58f8c546-d3c9-437b-af39-72fb0dfba1de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 17:03:44.407634 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:44.407608 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f8c546-d3c9-437b-af39-72fb0dfba1de-kube-api-access-sjmkh" (OuterVolumeSpecName: "kube-api-access-sjmkh") pod "58f8c546-d3c9-437b-af39-72fb0dfba1de" (UID: "58f8c546-d3c9-437b-af39-72fb0dfba1de"). InnerVolumeSpecName "kube-api-access-sjmkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:03:44.506264 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:44.506157 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjmkh\" (UniqueName: \"kubernetes.io/projected/58f8c546-d3c9-437b-af39-72fb0dfba1de-kube-api-access-sjmkh\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:44.506264 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:44.506186 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f8c546-d3c9-437b-af39-72fb0dfba1de-plugins-config-volume\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:44.697797 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:44.697758 2563 generic.go:356] "Generic (PLEG): container finished" podID="58f8c546-d3c9-437b-af39-72fb0dfba1de" containerID="af8ff4d3784a6a01303c8123235bff47cd691e552221291060fced0e4c655538" exitCode=0 Jul 26 17:03:44.697982 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:44.697802 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp" Jul 26 17:03:44.697982 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:44.697848 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp" event={"ID":"58f8c546-d3c9-437b-af39-72fb0dfba1de","Type":"ContainerDied","Data":"af8ff4d3784a6a01303c8123235bff47cd691e552221291060fced0e4c655538"} Jul 26 17:03:44.697982 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:44.697909 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp" event={"ID":"58f8c546-d3c9-437b-af39-72fb0dfba1de","Type":"ContainerDied","Data":"71dd19030bffabeb6101681bd2eba2971b7e0783b770dddc1af6ffc09cd30e12"} Jul 26 17:03:44.697982 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:44.697937 2563 scope.go:117] "RemoveContainer" containerID="af8ff4d3784a6a01303c8123235bff47cd691e552221291060fced0e4c655538" Jul 26 17:03:44.707151 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:44.707135 2563 scope.go:117] "RemoveContainer" containerID="af8ff4d3784a6a01303c8123235bff47cd691e552221291060fced0e4c655538" Jul 26 17:03:44.707431 ip-10-0-130-70 kubenswrapper[2563]: E0726 17:03:44.707415 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af8ff4d3784a6a01303c8123235bff47cd691e552221291060fced0e4c655538\": container with ID starting with af8ff4d3784a6a01303c8123235bff47cd691e552221291060fced0e4c655538 not found: ID does not exist" containerID="af8ff4d3784a6a01303c8123235bff47cd691e552221291060fced0e4c655538" Jul 26 17:03:44.707490 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:44.707438 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af8ff4d3784a6a01303c8123235bff47cd691e552221291060fced0e4c655538"} err="failed to get container status \"af8ff4d3784a6a01303c8123235bff47cd691e552221291060fced0e4c655538\": rpc error: code = NotFound desc = could not find container \"af8ff4d3784a6a01303c8123235bff47cd691e552221291060fced0e4c655538\": container with ID starting with af8ff4d3784a6a01303c8123235bff47cd691e552221291060fced0e4c655538 not found: ID does not exist" Jul 26 17:03:44.722633 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:44.722609 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp"] Jul 26 17:03:44.727510 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:44.727472 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1b2072-84b88f966b-v5rzp"] Jul 26 17:03:44.891210 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:44.891175 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58f8c546-d3c9-437b-af39-72fb0dfba1de" path="/var/lib/kubelet/pods/58f8c546-d3c9-437b-af39-72fb0dfba1de/volumes" Jul 26 17:03:54.488816 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:54.488782 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc"] Jul 26 17:03:54.489350 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:54.489057 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" podUID="a654d489-3ddf-4fd5-b428-f5cba87350c1" containerName="istio-proxy" containerID="cri-o://6747b18f0f76caa4729834be1210ea6ae02932bea1534b40bb4bd5f33399ef7d" gracePeriod=30 Jul 26 17:03:59.742845 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.742815 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 17:03:59.750903 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.750850 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-workload-certs\") pod \"a654d489-3ddf-4fd5-b428-f5cba87350c1\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " Jul 26 17:03:59.750903 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.750889 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-data\") pod \"a654d489-3ddf-4fd5-b428-f5cba87350c1\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " Jul 26 17:03:59.751061 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.750913 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-envoy\") pod \"a654d489-3ddf-4fd5-b428-f5cba87350c1\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " Jul 26 17:03:59.751061 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.750960 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a654d489-3ddf-4fd5-b428-f5cba87350c1-istiod-ca-cert\") pod \"a654d489-3ddf-4fd5-b428-f5cba87350c1\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " Jul 26 17:03:59.751061 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.750985 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-token\") pod \"a654d489-3ddf-4fd5-b428-f5cba87350c1\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " Jul 26 17:03:59.751061 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.751018 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-credential-socket\") pod \"a654d489-3ddf-4fd5-b428-f5cba87350c1\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " Jul 26 17:03:59.751061 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.751054 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-workload-socket\") pod \"a654d489-3ddf-4fd5-b428-f5cba87350c1\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " Jul 26 17:03:59.751342 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.751086 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vf8v7\" (UniqueName: \"kubernetes.io/projected/a654d489-3ddf-4fd5-b428-f5cba87350c1-kube-api-access-vf8v7\") pod \"a654d489-3ddf-4fd5-b428-f5cba87350c1\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " Jul 26 17:03:59.751342 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.751140 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-podinfo\") pod \"a654d489-3ddf-4fd5-b428-f5cba87350c1\" (UID: \"a654d489-3ddf-4fd5-b428-f5cba87350c1\") " Jul 26 17:03:59.751342 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.751159 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a654d489-3ddf-4fd5-b428-f5cba87350c1" (UID: "a654d489-3ddf-4fd5-b428-f5cba87350c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:03:59.751503 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.751321 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a654d489-3ddf-4fd5-b428-f5cba87350c1" (UID: "a654d489-3ddf-4fd5-b428-f5cba87350c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:03:59.751503 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.751370 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a654d489-3ddf-4fd5-b428-f5cba87350c1" (UID: "a654d489-3ddf-4fd5-b428-f5cba87350c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:03:59.751503 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.751423 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a654d489-3ddf-4fd5-b428-f5cba87350c1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a654d489-3ddf-4fd5-b428-f5cba87350c1" (UID: "a654d489-3ddf-4fd5-b428-f5cba87350c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 17:03:59.751656 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.751504 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-workload-certs\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:59.751656 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.751522 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-credential-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:59.751656 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.751536 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-workload-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:59.752213 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.752187 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-data" (OuterVolumeSpecName: "istio-data") pod "a654d489-3ddf-4fd5-b428-f5cba87350c1" (UID: "a654d489-3ddf-4fd5-b428-f5cba87350c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:03:59.753256 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.753212 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a654d489-3ddf-4fd5-b428-f5cba87350c1" (UID: "a654d489-3ddf-4fd5-b428-f5cba87350c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:03:59.753256 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.753210 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-token" (OuterVolumeSpecName: "istio-token") pod "a654d489-3ddf-4fd5-b428-f5cba87350c1" (UID: "a654d489-3ddf-4fd5-b428-f5cba87350c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:03:59.753381 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.753260 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a654d489-3ddf-4fd5-b428-f5cba87350c1-kube-api-access-vf8v7" (OuterVolumeSpecName: "kube-api-access-vf8v7") pod "a654d489-3ddf-4fd5-b428-f5cba87350c1" (UID: "a654d489-3ddf-4fd5-b428-f5cba87350c1"). InnerVolumeSpecName "kube-api-access-vf8v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:03:59.753426 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.753380 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a654d489-3ddf-4fd5-b428-f5cba87350c1" (UID: "a654d489-3ddf-4fd5-b428-f5cba87350c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 17:03:59.791054 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.791009 2563 generic.go:356] "Generic (PLEG): container finished" podID="a654d489-3ddf-4fd5-b428-f5cba87350c1" containerID="6747b18f0f76caa4729834be1210ea6ae02932bea1534b40bb4bd5f33399ef7d" exitCode=0 Jul 26 17:03:59.791258 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.791060 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" Jul 26 17:03:59.791258 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.791088 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" event={"ID":"a654d489-3ddf-4fd5-b428-f5cba87350c1","Type":"ContainerDied","Data":"6747b18f0f76caa4729834be1210ea6ae02932bea1534b40bb4bd5f33399ef7d"} Jul 26 17:03:59.791258 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.791144 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc" event={"ID":"a654d489-3ddf-4fd5-b428-f5cba87350c1","Type":"ContainerDied","Data":"3ad8d72ffe8f18791dcae8c5b7fd6321ff69838dc7c95576413cb7c0a2055e01"} Jul 26 17:03:59.791258 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.791174 2563 scope.go:117] "RemoveContainer" containerID="6747b18f0f76caa4729834be1210ea6ae02932bea1534b40bb4bd5f33399ef7d" Jul 26 17:03:59.800929 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.800912 2563 scope.go:117] "RemoveContainer" containerID="6aa3ee5275119b02877e894ebcd004a82a8b6a1957fd970d808bbda2174b6c54" Jul 26 17:03:59.808836 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.808824 2563 scope.go:117] "RemoveContainer" containerID="6747b18f0f76caa4729834be1210ea6ae02932bea1534b40bb4bd5f33399ef7d" Jul 26 17:03:59.809080 ip-10-0-130-70 kubenswrapper[2563]: E0726 17:03:59.809064 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6747b18f0f76caa4729834be1210ea6ae02932bea1534b40bb4bd5f33399ef7d\": container with ID starting with 6747b18f0f76caa4729834be1210ea6ae02932bea1534b40bb4bd5f33399ef7d not found: ID does not exist" containerID="6747b18f0f76caa4729834be1210ea6ae02932bea1534b40bb4bd5f33399ef7d" Jul 26 17:03:59.809128 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.809088 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6747b18f0f76caa4729834be1210ea6ae02932bea1534b40bb4bd5f33399ef7d"} err="failed to get container status \"6747b18f0f76caa4729834be1210ea6ae02932bea1534b40bb4bd5f33399ef7d\": rpc error: code = NotFound desc = could not find container \"6747b18f0f76caa4729834be1210ea6ae02932bea1534b40bb4bd5f33399ef7d\": container with ID starting with 6747b18f0f76caa4729834be1210ea6ae02932bea1534b40bb4bd5f33399ef7d not found: ID does not exist" Jul 26 17:03:59.809128 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.809103 2563 scope.go:117] "RemoveContainer" containerID="6aa3ee5275119b02877e894ebcd004a82a8b6a1957fd970d808bbda2174b6c54" Jul 26 17:03:59.809366 ip-10-0-130-70 kubenswrapper[2563]: E0726 17:03:59.809350 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa3ee5275119b02877e894ebcd004a82a8b6a1957fd970d808bbda2174b6c54\": container with ID starting with 6aa3ee5275119b02877e894ebcd004a82a8b6a1957fd970d808bbda2174b6c54 not found: ID does not exist" containerID="6aa3ee5275119b02877e894ebcd004a82a8b6a1957fd970d808bbda2174b6c54" Jul 26 17:03:59.809423 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.809370 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa3ee5275119b02877e894ebcd004a82a8b6a1957fd970d808bbda2174b6c54"} err="failed to get container status \"6aa3ee5275119b02877e894ebcd004a82a8b6a1957fd970d808bbda2174b6c54\": rpc error: code = NotFound desc = could not find container \"6aa3ee5275119b02877e894ebcd004a82a8b6a1957fd970d808bbda2174b6c54\": container with ID starting with 6aa3ee5275119b02877e894ebcd004a82a8b6a1957fd970d808bbda2174b6c54 not found: ID does not exist" Jul 26 17:03:59.814555 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.814534 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc"] Jul 26 17:03:59.820744 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.820722 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-1b2072-openshift-default-54bd94df9b-tjrvc"] Jul 26 17:03:59.852003 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.851978 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-podinfo\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:59.852003 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.852000 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-data\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:59.852003 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.852009 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-envoy\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:59.852181 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.852016 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a654d489-3ddf-4fd5-b428-f5cba87350c1-istiod-ca-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:59.852181 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.852024 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a654d489-3ddf-4fd5-b428-f5cba87350c1-istio-token\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:03:59.852181 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:03:59.852034 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf8v7\" (UniqueName: \"kubernetes.io/projected/a654d489-3ddf-4fd5-b428-f5cba87350c1-kube-api-access-vf8v7\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:04:00.891076 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:00.891042 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a654d489-3ddf-4fd5-b428-f5cba87350c1" path="/var/lib/kubelet/pods/a654d489-3ddf-4fd5-b428-f5cba87350c1/volumes" Jul 26 17:04:05.628436 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:05.628398 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj"] Jul 26 17:04:05.628938 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:05.628637 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" podUID="f45c3dcb-dd50-471a-a8d4-b5fc18848a04" containerName="maas-api" containerID="cri-o://08404b0c7f4c63dd553e19fce5eb6616872a8bb85d2cf4ed93137657d5625eeb" gracePeriod=30 Jul 26 17:04:05.825624 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:05.825592 2563 generic.go:356] "Generic (PLEG): container finished" podID="f45c3dcb-dd50-471a-a8d4-b5fc18848a04" containerID="08404b0c7f4c63dd553e19fce5eb6616872a8bb85d2cf4ed93137657d5625eeb" exitCode=0 Jul 26 17:04:05.825799 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:05.825671 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" event={"ID":"f45c3dcb-dd50-471a-a8d4-b5fc18848a04","Type":"ContainerDied","Data":"08404b0c7f4c63dd553e19fce5eb6616872a8bb85d2cf4ed93137657d5625eeb"} Jul 26 17:04:05.891304 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:05.891235 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" Jul 26 17:04:05.908239 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:05.908188 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg7mx\" (UniqueName: \"kubernetes.io/projected/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-kube-api-access-tg7mx\") pod \"f45c3dcb-dd50-471a-a8d4-b5fc18848a04\" (UID: \"f45c3dcb-dd50-471a-a8d4-b5fc18848a04\") " Jul 26 17:04:05.908422 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:05.908263 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-openshift-service-ca\") pod \"f45c3dcb-dd50-471a-a8d4-b5fc18848a04\" (UID: \"f45c3dcb-dd50-471a-a8d4-b5fc18848a04\") " Jul 26 17:04:05.908422 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:05.908399 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-maas-api-tls\") pod \"f45c3dcb-dd50-471a-a8d4-b5fc18848a04\" (UID: \"f45c3dcb-dd50-471a-a8d4-b5fc18848a04\") " Jul 26 17:04:05.908629 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:05.908603 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f45c3dcb-dd50-471a-a8d4-b5fc18848a04" (UID: "f45c3dcb-dd50-471a-a8d4-b5fc18848a04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 17:04:05.908799 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:05.908765 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-openshift-service-ca\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:04:05.910897 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:05.910861 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f45c3dcb-dd50-471a-a8d4-b5fc18848a04" (UID: "f45c3dcb-dd50-471a-a8d4-b5fc18848a04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 17:04:05.911044 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:05.911015 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-kube-api-access-tg7mx" (OuterVolumeSpecName: "kube-api-access-tg7mx") pod "f45c3dcb-dd50-471a-a8d4-b5fc18848a04" (UID: "f45c3dcb-dd50-471a-a8d4-b5fc18848a04"). InnerVolumeSpecName "kube-api-access-tg7mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:04:06.009401 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:06.009369 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-maas-api-tls\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:04:06.009401 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:06.009397 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg7mx\" (UniqueName: \"kubernetes.io/projected/f45c3dcb-dd50-471a-a8d4-b5fc18848a04-kube-api-access-tg7mx\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:04:06.839415 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:06.839379 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" event={"ID":"f45c3dcb-dd50-471a-a8d4-b5fc18848a04","Type":"ContainerDied","Data":"dd31c8308c3e02ace90e9f54835473f42c278ad859d77094737d59513a3d8c18"} Jul 26 17:04:06.839415 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:06.839404 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj" Jul 26 17:04:06.839893 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:06.839463 2563 scope.go:117] "RemoveContainer" containerID="08404b0c7f4c63dd553e19fce5eb6616872a8bb85d2cf4ed93137657d5625eeb" Jul 26 17:04:06.862356 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:06.862330 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj"] Jul 26 17:04:06.867110 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:06.867081 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a06722-6d6d8865c4-6h6jj"] Jul 26 17:04:06.890762 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:06.890735 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f45c3dcb-dd50-471a-a8d4-b5fc18848a04" path="/var/lib/kubelet/pods/f45c3dcb-dd50-471a-a8d4-b5fc18848a04/volumes" Jul 26 17:04:16.352348 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:16.352267 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m"] Jul 26 17:04:16.352852 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:16.352528 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" podUID="3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" containerName="istio-proxy" containerID="cri-o://f4b2f0f0060fe43a2b587a719ecec671ef27404ebe2468da7d30dd2bea03da07" gracePeriod=30 Jul 26 17:04:21.595374 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.595344 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 17:04:21.654023 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.653936 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlm9x\" (UniqueName: \"kubernetes.io/projected/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-kube-api-access-hlm9x\") pod \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " Jul 26 17:04:21.654185 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.654026 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istiod-ca-cert\") pod \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " Jul 26 17:04:21.654185 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.654059 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-token\") pod \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " Jul 26 17:04:21.654185 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.654093 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-podinfo\") pod \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " Jul 26 17:04:21.654185 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.654123 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-workload-socket\") pod \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " Jul 26 17:04:21.654185 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.654178 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-envoy\") pod \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " Jul 26 17:04:21.654488 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.654206 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-data\") pod \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " Jul 26 17:04:21.654488 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.654280 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-workload-certs\") pod \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " Jul 26 17:04:21.654488 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.654335 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-credential-socket\") pod \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\" (UID: \"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb\") " Jul 26 17:04:21.654488 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.654452 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" (UID: "3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:04:21.654710 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.654565 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" (UID: "3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 17:04:21.654710 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.654691 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" (UID: "3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:04:21.654710 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.654703 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istiod-ca-cert\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:04:21.654873 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.654719 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-workload-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:04:21.654873 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.654788 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" (UID: "3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:04:21.655188 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.655162 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-data" (OuterVolumeSpecName: "istio-data") pod "3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" (UID: "3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:04:21.656355 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.656330 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-token" (OuterVolumeSpecName: "istio-token") pod "3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" (UID: "3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:04:21.656680 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.656653 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" (UID: "3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:04:21.656824 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.656803 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" (UID: "3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 17:04:21.656911 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.656893 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-kube-api-access-hlm9x" (OuterVolumeSpecName: "kube-api-access-hlm9x") pod "3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" (UID: "3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb"). InnerVolumeSpecName "kube-api-access-hlm9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:04:21.755338 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.755301 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-workload-certs\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:04:21.755338 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.755333 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-credential-socket\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:04:21.755338 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.755344 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlm9x\" (UniqueName: \"kubernetes.io/projected/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-kube-api-access-hlm9x\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:04:21.755586 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.755354 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-token\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:04:21.755586 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.755363 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-podinfo\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:04:21.755586 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.755371 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-envoy\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:04:21.755586 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.755378 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb-istio-data\") on node \"ip-10-0-130-70.ec2.internal\" DevicePath \"\"" Jul 26 17:04:21.920464 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.920373 2563 generic.go:356] "Generic (PLEG): container finished" podID="3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" containerID="f4b2f0f0060fe43a2b587a719ecec671ef27404ebe2468da7d30dd2bea03da07" exitCode=0 Jul 26 17:04:21.920464 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.920422 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" Jul 26 17:04:21.920647 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.920473 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" event={"ID":"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb","Type":"ContainerDied","Data":"f4b2f0f0060fe43a2b587a719ecec671ef27404ebe2468da7d30dd2bea03da07"} Jul 26 17:04:21.920647 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.920532 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m" event={"ID":"3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb","Type":"ContainerDied","Data":"8e2afbc163801a32f641f13c4a086b0d3311043eea633894cb848f17da7ef6d1"} Jul 26 17:04:21.920647 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.920561 2563 scope.go:117] "RemoveContainer" containerID="f4b2f0f0060fe43a2b587a719ecec671ef27404ebe2468da7d30dd2bea03da07" Jul 26 17:04:21.930231 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.930199 2563 scope.go:117] "RemoveContainer" containerID="aafed93bb17c5f56f10ccf3ac44ce0bed06263306b49ef0acce4e17ad1fc9521" Jul 26 17:04:21.938236 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.938207 2563 scope.go:117] "RemoveContainer" containerID="f4b2f0f0060fe43a2b587a719ecec671ef27404ebe2468da7d30dd2bea03da07" Jul 26 17:04:21.938496 ip-10-0-130-70 kubenswrapper[2563]: E0726 17:04:21.938474 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b2f0f0060fe43a2b587a719ecec671ef27404ebe2468da7d30dd2bea03da07\": container with ID starting with f4b2f0f0060fe43a2b587a719ecec671ef27404ebe2468da7d30dd2bea03da07 not found: ID does not exist" containerID="f4b2f0f0060fe43a2b587a719ecec671ef27404ebe2468da7d30dd2bea03da07" Jul 26 17:04:21.938540 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.938505 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b2f0f0060fe43a2b587a719ecec671ef27404ebe2468da7d30dd2bea03da07"} err="failed to get container status \"f4b2f0f0060fe43a2b587a719ecec671ef27404ebe2468da7d30dd2bea03da07\": rpc error: code = NotFound desc = could not find container \"f4b2f0f0060fe43a2b587a719ecec671ef27404ebe2468da7d30dd2bea03da07\": container with ID starting with f4b2f0f0060fe43a2b587a719ecec671ef27404ebe2468da7d30dd2bea03da07 not found: ID does not exist" Jul 26 17:04:21.938540 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.938523 2563 scope.go:117] "RemoveContainer" containerID="aafed93bb17c5f56f10ccf3ac44ce0bed06263306b49ef0acce4e17ad1fc9521" Jul 26 17:04:21.938762 ip-10-0-130-70 kubenswrapper[2563]: E0726 17:04:21.938741 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aafed93bb17c5f56f10ccf3ac44ce0bed06263306b49ef0acce4e17ad1fc9521\": container with ID starting with aafed93bb17c5f56f10ccf3ac44ce0bed06263306b49ef0acce4e17ad1fc9521 not found: ID does not exist" containerID="aafed93bb17c5f56f10ccf3ac44ce0bed06263306b49ef0acce4e17ad1fc9521" Jul 26 17:04:21.938807 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.938768 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aafed93bb17c5f56f10ccf3ac44ce0bed06263306b49ef0acce4e17ad1fc9521"} err="failed to get container status \"aafed93bb17c5f56f10ccf3ac44ce0bed06263306b49ef0acce4e17ad1fc9521\": rpc error: code = NotFound desc = could not find container \"aafed93bb17c5f56f10ccf3ac44ce0bed06263306b49ef0acce4e17ad1fc9521\": container with ID starting with aafed93bb17c5f56f10ccf3ac44ce0bed06263306b49ef0acce4e17ad1fc9521 not found: ID does not exist" Jul 26 17:04:21.942305 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.942283 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m"] Jul 26 17:04:21.951365 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:21.949068 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-a06722-openshift-default-779bd68dd8-2tf5m"] Jul 26 17:04:22.474450 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:22.474412 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5656c79bb6-gprrd_db312ba1-76de-4f39-b952-d32b58c8306c/manager/0.log" Jul 26 17:04:22.891186 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:22.891149 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" path="/var/lib/kubelet/pods/3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb/volumes" Jul 26 17:04:22.931576 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:22.931546 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bd4986cdb-kg7w6_5b87e136-eb08-4ca1-92f6-1b6e7d722248/copy-manifests/0.log" Jul 26 17:04:22.945194 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:22.945164 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bd4986cdb-kg7w6_5b87e136-eb08-4ca1-92f6-1b6e7d722248/manager/0.log" Jul 26 17:04:23.416882 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:23.416851 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-545d4d4db9-5sbqx_f9a4b6e5-6baf-4930-9329-26ea2abb176c/maas-api/0.log" Jul 26 17:04:24.612084 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:24.612056 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2_8ae15d55-d75e-478e-beb6-044d763f1de6/util/0.log" Jul 26 17:04:24.618560 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:24.618532 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2_8ae15d55-d75e-478e-beb6-044d763f1de6/pull/0.log" Jul 26 17:04:24.624108 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:24.624090 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2_8ae15d55-d75e-478e-beb6-044d763f1de6/extract/0.log" Jul 26 17:04:24.732874 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:24.732842 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z_2c801712-0ec8-41d7-ba1a-6e0616e58361/util/0.log" Jul 26 17:04:24.741509 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:24.741483 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z_2c801712-0ec8-41d7-ba1a-6e0616e58361/pull/0.log" Jul 26 17:04:24.747644 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:24.747622 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z_2c801712-0ec8-41d7-ba1a-6e0616e58361/extract/0.log" Jul 26 17:04:24.859819 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:24.859790 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q_5603a10c-7a12-4635-a095-9db354be05b8/util/0.log" Jul 26 17:04:24.866361 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:24.866298 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q_5603a10c-7a12-4635-a095-9db354be05b8/pull/0.log" Jul 26 17:04:24.873068 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:24.873050 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q_5603a10c-7a12-4635-a095-9db354be05b8/extract/0.log" Jul 26 17:04:24.982566 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:24.982532 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp_8369e121-2656-4b33-a7de-53e136a71530/util/0.log" Jul 26 17:04:24.988519 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:24.988497 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp_8369e121-2656-4b33-a7de-53e136a71530/pull/0.log" Jul 26 17:04:24.994698 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:24.994683 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp_8369e121-2656-4b33-a7de-53e136a71530/extract/0.log" Jul 26 17:04:25.675841 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:25.675811 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-g2g6l_0124524d-8ad5-4512-963a-af28513c8116/manager/0.log" Jul 26 17:04:26.233857 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:26.233824 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj_4c09d668-aa2a-494f-902c-2c53c190bfc6/istio-proxy/0.log" Jul 26 17:04:26.349575 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:26.349539 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8g8p9_c6f4215e-9b81-4a1e-8c75-68edb9616df2/discovery/0.log" Jul 26 17:04:27.600571 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:27.600540 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-4vk72_52f8bd5b-4290-4648-b305-4c478c96744f/main/0.log" Jul 26 17:04:36.206919 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:36.206884 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hs2qb_9f8db869-87b0-4a2c-b292-b78f22f3a391/global-pull-secret-syncer/0.log" Jul 26 17:04:36.347866 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:36.347818 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l7cw2_f7f65fa5-91ab-49cf-ab27-f4af947b3bbe/konnectivity-agent/0.log" Jul 26 17:04:36.403202 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:36.403173 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-70.ec2.internal_e33f5784a8667eeea92ef51af7a40e20/haproxy/0.log" Jul 26 17:04:40.617480 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:40.617449 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2_8ae15d55-d75e-478e-beb6-044d763f1de6/extract/0.log" Jul 26 17:04:40.641543 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:40.641510 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2_8ae15d55-d75e-478e-beb6-044d763f1de6/util/0.log" Jul 26 17:04:40.669230 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:40.669191 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759j69b2_8ae15d55-d75e-478e-beb6-044d763f1de6/pull/0.log" Jul 26 17:04:40.715754 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:40.715727 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z_2c801712-0ec8-41d7-ba1a-6e0616e58361/extract/0.log" Jul 26 17:04:40.739052 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:40.739024 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z_2c801712-0ec8-41d7-ba1a-6e0616e58361/util/0.log" Jul 26 17:04:40.762008 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:40.761986 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e044c2z_2c801712-0ec8-41d7-ba1a-6e0616e58361/pull/0.log" Jul 26 17:04:40.789005 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:40.788983 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q_5603a10c-7a12-4635-a095-9db354be05b8/extract/0.log" Jul 26 17:04:40.815810 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:40.815786 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q_5603a10c-7a12-4635-a095-9db354be05b8/util/0.log" Jul 26 17:04:40.839928 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:40.839902 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8g8q_5603a10c-7a12-4635-a095-9db354be05b8/pull/0.log" Jul 26 17:04:40.869088 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:40.869025 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp_8369e121-2656-4b33-a7de-53e136a71530/extract/0.log" Jul 26 17:04:40.890815 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:40.890789 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp_8369e121-2656-4b33-a7de-53e136a71530/util/0.log" Jul 26 17:04:40.912860 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:40.912828 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sprlp_8369e121-2656-4b33-a7de-53e136a71530/pull/0.log" Jul 26 17:04:41.589510 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:41.589472 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-g2g6l_0124524d-8ad5-4512-963a-af28513c8116/manager/0.log" Jul 26 17:04:43.479522 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:43.479483 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ec858743-e2e3-4428-8b5a-07e1c4f4a1b3/alertmanager/0.log" Jul 26 17:04:43.516421 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:43.516317 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ec858743-e2e3-4428-8b5a-07e1c4f4a1b3/config-reloader/0.log" Jul 26 17:04:43.542651 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:43.542616 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ec858743-e2e3-4428-8b5a-07e1c4f4a1b3/kube-rbac-proxy-web/0.log" Jul 26 17:04:43.567307 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:43.567281 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ec858743-e2e3-4428-8b5a-07e1c4f4a1b3/kube-rbac-proxy/0.log" Jul 26 17:04:43.590553 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:43.590525 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ec858743-e2e3-4428-8b5a-07e1c4f4a1b3/kube-rbac-proxy-metric/0.log" Jul 26 17:04:43.614447 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:43.614420 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ec858743-e2e3-4428-8b5a-07e1c4f4a1b3/prom-label-proxy/0.log" Jul 26 17:04:43.639251 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:43.639212 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ec858743-e2e3-4428-8b5a-07e1c4f4a1b3/init-config-reloader/0.log" Jul 26 17:04:43.716265 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:43.716216 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-cjzbh_4cd9bc19-660a-4d27-9af5-b34498225894/kube-state-metrics/0.log" Jul 26 17:04:43.745551 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:43.745523 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-cjzbh_4cd9bc19-660a-4d27-9af5-b34498225894/kube-rbac-proxy-main/0.log" Jul 26 17:04:43.771408 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:43.771339 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-cjzbh_4cd9bc19-660a-4d27-9af5-b34498225894/kube-rbac-proxy-self/0.log" Jul 26 17:04:43.835321 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:43.835293 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-jmztv_e9a61391-e4db-44b4-900b-4d544bc9f39f/monitoring-plugin/0.log" Jul 26 17:04:43.979084 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:43.979051 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dcds8_15339a8c-492a-4a85-9988-59d8942ae27d/node-exporter/0.log" Jul 26 17:04:44.011617 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:44.011591 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dcds8_15339a8c-492a-4a85-9988-59d8942ae27d/kube-rbac-proxy/0.log" Jul 26 17:04:44.036396 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:44.036374 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dcds8_15339a8c-492a-4a85-9988-59d8942ae27d/init-textfile/0.log" Jul 26 17:04:44.142858 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:44.142819 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-4h5nf_6ffc8cc6-21c2-4827-959f-25d6cfefffa2/kube-rbac-proxy-main/0.log" Jul 26 17:04:44.169436 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:44.169411 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-4h5nf_6ffc8cc6-21c2-4827-959f-25d6cfefffa2/kube-rbac-proxy-self/0.log" Jul 26 17:04:44.195323 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:44.195279 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-4h5nf_6ffc8cc6-21c2-4827-959f-25d6cfefffa2/openshift-state-metrics/0.log" Jul 26 17:04:44.487905 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:44.487824 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b967cd444-tk24s_807200d8-49e0-4189-9ea5-4d4fd6b2e46c/telemeter-client/0.log" Jul 26 17:04:44.509955 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:44.509917 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b967cd444-tk24s_807200d8-49e0-4189-9ea5-4d4fd6b2e46c/reload/0.log" Jul 26 17:04:44.531903 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:44.531884 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b967cd444-tk24s_807200d8-49e0-4189-9ea5-4d4fd6b2e46c/kube-rbac-proxy/0.log" Jul 26 17:04:46.044812 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.044774 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc"] Jul 26 17:04:46.045309 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045289 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96" containerName="payload-processing" Jul 26 17:04:46.045309 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045309 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96" containerName="payload-processing" Jul 26 17:04:46.045458 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045327 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a654d489-3ddf-4fd5-b428-f5cba87350c1" containerName="istio-proxy" Jul 26 17:04:46.045458 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045335 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a654d489-3ddf-4fd5-b428-f5cba87350c1" containerName="istio-proxy" Jul 26 17:04:46.045458 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045347 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="643f5bc0-8886-4afb-8f0f-c2d1de1b59a2" containerName="payload-pre-processing" Jul 26 17:04:46.045458 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045355 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="643f5bc0-8886-4afb-8f0f-c2d1de1b59a2" containerName="payload-pre-processing" Jul 26 17:04:46.045458 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045368 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322dd3c3-3d4e-46e9-bccc-5e599d3812bb" containerName="revoke-keys" Jul 26 17:04:46.045458 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045377 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="322dd3c3-3d4e-46e9-bccc-5e599d3812bb" containerName="revoke-keys" Jul 26 17:04:46.045458 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045393 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a654d489-3ddf-4fd5-b428-f5cba87350c1" containerName="istio-proxy" Jul 26 17:04:46.045458 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045402 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a654d489-3ddf-4fd5-b428-f5cba87350c1" containerName="istio-proxy" Jul 26 17:04:46.045458 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045411 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b94162c9-2e19-4b46-ae85-9a420bda7212" containerName="istio-proxy" Jul 26 17:04:46.045458 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045419 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b94162c9-2e19-4b46-ae85-9a420bda7212" containerName="istio-proxy" Jul 26 17:04:46.045458 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045431 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab62e80f-a691-4596-9098-1079a14de9ed" containerName="istio-proxy" Jul 26 17:04:46.045458 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045439 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab62e80f-a691-4596-9098-1079a14de9ed" containerName="istio-proxy" Jul 26 17:04:46.045458 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045453 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" containerName="istio-proxy" Jul 26 17:04:46.045458 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045462 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" containerName="istio-proxy" Jul 26 17:04:46.046156 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045473 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58f8c546-d3c9-437b-af39-72fb0dfba1de" containerName="payload-processing" Jul 26 17:04:46.046156 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045481 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="58f8c546-d3c9-437b-af39-72fb0dfba1de" containerName="payload-processing" Jul 26 17:04:46.046156 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045496 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" containerName="istio-proxy" Jul 26 17:04:46.046156 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045504 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" containerName="istio-proxy" Jul 26 17:04:46.046156 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045516 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f45c3dcb-dd50-471a-a8d4-b5fc18848a04" containerName="maas-api" Jul 26 17:04:46.046156 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045524 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f45c3dcb-dd50-471a-a8d4-b5fc18848a04" containerName="maas-api" Jul 26 17:04:46.046156 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045624 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a654d489-3ddf-4fd5-b428-f5cba87350c1" containerName="istio-proxy" Jul 26 17:04:46.046156 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045636 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b94162c9-2e19-4b46-ae85-9a420bda7212" containerName="istio-proxy" Jul 26 17:04:46.046156 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045647 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" containerName="istio-proxy" Jul 26 17:04:46.046156 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045656 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="58f8c546-d3c9-437b-af39-72fb0dfba1de" containerName="payload-processing" Jul 26 17:04:46.046156 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045665 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b94162c9-2e19-4b46-ae85-9a420bda7212" containerName="istio-proxy" Jul 26 17:04:46.046156 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045673 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="643f5bc0-8886-4afb-8f0f-c2d1de1b59a2" containerName="payload-pre-processing" Jul 26 17:04:46.046156 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045682 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="322dd3c3-3d4e-46e9-bccc-5e599d3812bb" containerName="revoke-keys" Jul 26 17:04:46.046156 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045695 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab62e80f-a691-4596-9098-1079a14de9ed" containerName="istio-proxy" Jul 26 17:04:46.046156 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045704 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f45c3dcb-dd50-471a-a8d4-b5fc18848a04" containerName="maas-api" Jul 26 17:04:46.046156 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045712 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f0c2eb8-e9ae-4693-92b8-dccb9c1ada96" containerName="payload-processing" Jul 26 17:04:46.046156 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045721 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab62e80f-a691-4596-9098-1079a14de9ed" containerName="istio-proxy" Jul 26 17:04:46.046156 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.045728 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c0ccea9-8ba4-4d7b-94de-8378bdf67fbb" containerName="istio-proxy" Jul 26 17:04:46.049368 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.049346 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:46.054209 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.054192 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bnlmg\"/\"default-dockercfg-g2bhf\"" Jul 26 17:04:46.055763 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.055387 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bnlmg\"/\"openshift-service-ca.crt\"" Jul 26 17:04:46.055763 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.055429 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bnlmg\"/\"kube-root-ca.crt\"" Jul 26 17:04:46.070053 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.070026 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc"] Jul 26 17:04:46.076805 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.076779 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7-lib-modules\") pod \"perf-node-gather-daemonset-lk8zc\" (UID: \"d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7\") " pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:46.076916 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.076814 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scx9c\" (UniqueName: \"kubernetes.io/projected/d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7-kube-api-access-scx9c\") pod \"perf-node-gather-daemonset-lk8zc\" (UID: \"d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7\") " pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:46.076916 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.076856 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7-sys\") pod \"perf-node-gather-daemonset-lk8zc\" (UID: \"d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7\") " pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:46.077031 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.076942 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7-proc\") pod \"perf-node-gather-daemonset-lk8zc\" (UID: \"d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7\") " pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:46.077031 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.076996 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7-podres\") pod \"perf-node-gather-daemonset-lk8zc\" (UID: \"d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7\") " pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:46.178053 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.178015 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7-proc\") pod \"perf-node-gather-daemonset-lk8zc\" (UID: \"d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7\") " pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:46.178284 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.178068 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7-podres\") pod \"perf-node-gather-daemonset-lk8zc\" (UID: \"d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7\") " pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:46.178284 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.178117 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7-lib-modules\") pod \"perf-node-gather-daemonset-lk8zc\" (UID: \"d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7\") " pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:46.178284 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.178140 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scx9c\" (UniqueName: \"kubernetes.io/projected/d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7-kube-api-access-scx9c\") pod \"perf-node-gather-daemonset-lk8zc\" (UID: \"d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7\") " pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:46.178284 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.178144 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7-proc\") pod \"perf-node-gather-daemonset-lk8zc\" (UID: \"d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7\") " pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:46.178284 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.178178 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7-sys\") pod \"perf-node-gather-daemonset-lk8zc\" (UID: \"d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7\") " pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:46.178284 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.178267 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7-podres\") pod \"perf-node-gather-daemonset-lk8zc\" (UID: \"d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7\") " pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:46.178527 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.178285 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7-lib-modules\") pod \"perf-node-gather-daemonset-lk8zc\" (UID: \"d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7\") " pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:46.178527 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.178286 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7-sys\") pod \"perf-node-gather-daemonset-lk8zc\" (UID: \"d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7\") " pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:46.187282 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.187264 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scx9c\" (UniqueName: \"kubernetes.io/projected/d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7-kube-api-access-scx9c\") pod \"perf-node-gather-daemonset-lk8zc\" (UID: \"d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7\") " pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:46.359111 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.359021 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:46.489591 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:46.489565 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc"] Jul 26 17:04:46.492182 ip-10-0-130-70 kubenswrapper[2563]: W0726 17:04:46.492156 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd9b012b5_163d_43c1_8e2e_5b23ac8fa3b7.slice/crio-d08846c4764d5538d0c15cefb8308a1087b0e9c83f918bbf8844af52f0e27560 WatchSource:0}: Error finding container d08846c4764d5538d0c15cefb8308a1087b0e9c83f918bbf8844af52f0e27560: Status 404 returned error can't find the container with id d08846c4764d5538d0c15cefb8308a1087b0e9c83f918bbf8844af52f0e27560 Jul 26 17:04:47.048404 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:47.048362 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" event={"ID":"d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7","Type":"ContainerStarted","Data":"95fc0ccb50e1ff815947a8695f61fad9f2c1d6a2a1b8983c38030a70be666c46"} Jul 26 17:04:47.048812 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:47.048433 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" event={"ID":"d9b012b5-163d-43c1-8e2e-5b23ac8fa3b7","Type":"ContainerStarted","Data":"d08846c4764d5538d0c15cefb8308a1087b0e9c83f918bbf8844af52f0e27560"} Jul 26 17:04:47.048812 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:47.048462 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:47.067951 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:47.067902 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" podStartSLOduration=1.067886853 podStartE2EDuration="1.067886853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 17:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:04:47.065653658 +0000 UTC m=+3360.646740232" watchObservedRunningTime="2026-07-26 17:04:47.067886853 +0000 UTC m=+3360.648973423" Jul 26 17:04:47.143096 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:47.143066 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-76697d598c-pclxb_a48906d8-915f-4211-9319-5b5786f92eee/console/0.log" Jul 26 17:04:48.874303 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:48.874272 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b27r6_3cee3f58-a577-48df-9b42-ea4fba7ed3cf/dns/0.log" Jul 26 17:04:48.898945 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:48.898920 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b27r6_3cee3f58-a577-48df-9b42-ea4fba7ed3cf/kube-rbac-proxy/0.log" Jul 26 17:04:49.051750 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:49.051715 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bqnx2_8b9ae82e-3ce4-4c7f-9902-a1ef79faae1f/dns-node-resolver/0.log" Jul 26 17:04:49.643546 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:49.643512 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2nwxs_e8fd86d4-e511-4f8e-962d-8c6726ba4cd1/node-ca/0.log" Jul 26 17:04:50.682190 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:50.682163 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj_4c09d668-aa2a-494f-902c-2c53c190bfc6/istio-proxy/0.log" Jul 26 17:04:50.844467 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:50.844434 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8g8p9_c6f4215e-9b81-4a1e-8c75-68edb9616df2/discovery/0.log" Jul 26 17:04:51.909347 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:51.909314 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hsc48_62ca1095-ccae-444f-b8e6-2625663e191f/serve-healthcheck-canary/0.log" Jul 26 17:04:52.451872 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:52.451840 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2sr94_c2c98da9-5848-4b54-8034-d60ef7e81fcd/kube-rbac-proxy/0.log" Jul 26 17:04:52.474073 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:52.474046 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2sr94_c2c98da9-5848-4b54-8034-d60ef7e81fcd/exporter/0.log" Jul 26 17:04:52.496157 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:52.496138 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2sr94_c2c98da9-5848-4b54-8034-d60ef7e81fcd/extractor/0.log" Jul 26 17:04:53.063148 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:53.063118 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bnlmg/perf-node-gather-daemonset-lk8zc" Jul 26 17:04:55.325749 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:55.325710 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5656c79bb6-gprrd_db312ba1-76de-4f39-b952-d32b58c8306c/manager/0.log" Jul 26 17:04:55.624753 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:55.624724 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bd4986cdb-kg7w6_5b87e136-eb08-4ca1-92f6-1b6e7d722248/manager/0.log" Jul 26 17:04:55.646317 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:55.646243 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bd4986cdb-kg7w6_5b87e136-eb08-4ca1-92f6-1b6e7d722248/copy-manifests/0.log" Jul 26 17:04:56.652162 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:56.652129 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj_4c09d668-aa2a-494f-902c-2c53c190bfc6/istio-proxy/0.log" Jul 26 17:04:56.801366 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:56.801335 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8g8p9_c6f4215e-9b81-4a1e-8c75-68edb9616df2/discovery/0.log" Jul 26 17:04:57.114862 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:57.114837 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-76c849bdf9-8cqr5_48d6169c-7e2b-4578-a4cc-b5ab9e6a9292/manager/0.log" Jul 26 17:04:58.391503 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:58.391476 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cgrbrj_4c09d668-aa2a-494f-902c-2c53c190bfc6/istio-proxy/0.log" Jul 26 17:04:58.540933 ip-10-0-130-70 kubenswrapper[2563]: I0726 17:04:58.540900 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8g8p9_c6f4215e-9b81-4a1e-8c75-68edb9616df2/discovery/0.log"