Jul 14 19:10:46.751908 ip-10-0-128-74 systemd[1]: Starting Kubernetes Kubelet... Jul 14 19:10:47.235040 ip-10-0-128-74 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 19:10:47.235040 ip-10-0-128-74 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 19:10:47.235040 ip-10-0-128-74 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 19:10:47.235040 ip-10-0-128-74 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 19:10:47.235040 ip-10-0-128-74 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 19:10:47.238210 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.238036 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 19:10:47.244492 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244461 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:10:47.244492 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244487 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:10:47.244492 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244491 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:10:47.244492 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244495 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:10:47.244492 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244498 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:10:47.244492 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244501 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244510 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244513 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244517 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244520 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244523 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244526 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244530 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244533 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244536 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244540 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244543 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244546 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244550 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244555 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244558 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244562 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244565 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244568 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244571 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:10:47.244778 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244574 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244577 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244581 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244586 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244589 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244592 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244598 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244601 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244604 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244607 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244609 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244612 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244616 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244619 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244623 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244625 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244628 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244631 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244634 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:10:47.245286 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244656 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244661 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244664 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244667 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244678 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244681 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244684 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244687 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244690 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244693 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244696 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244699 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244704 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244707 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244710 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244713 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244716 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244719 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244722 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244725 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:10:47.245807 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244728 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244731 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244733 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244736 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244739 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244744 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244747 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244752 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244755 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244758 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244765 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244769 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244773 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244777 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244780 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244784 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244790 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244800 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244803 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:10:47.246314 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244806 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244809 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244812 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244815 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244818 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244830 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244833 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244836 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244839 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244842 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244847 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244851 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244853 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244856 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244859 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.244862 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246209 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246216 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246220 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:10:47.246823 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246224 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246229 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246233 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246236 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246239 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246242 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246245 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246251 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246254 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246256 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246259 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246262 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246272 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246275 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246278 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246281 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246284 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246287 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246290 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246295 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:10:47.247307 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246298 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246301 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246304 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246307 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246310 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246312 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246315 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246318 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246321 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246324 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246328 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246332 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246337 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246341 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246344 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246347 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246350 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246352 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246355 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246358 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:10:47.247918 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246361 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246364 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246367 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246370 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246373 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246384 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246387 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246392 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246396 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246399 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246403 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246406 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246409 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246412 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246415 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246418 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246420 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246427 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246430 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:10:47.248441 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246433 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246436 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246439 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246442 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246445 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246449 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246454 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246458 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246461 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246464 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246470 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246475 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246478 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246480 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246484 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246486 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246489 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246492 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246497 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:10:47.248965 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246506 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246509 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246515 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246518 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246527 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246530 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246534 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246536 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246539 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246542 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246545 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246548 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246551 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246554 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246557 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246563 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246565 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.246568 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247337 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247344 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:10:47.249484 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247347 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247351 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247354 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247357 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247360 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247363 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247369 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247372 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247375 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247378 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247380 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247383 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247392 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247396 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247398 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247401 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247404 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247407 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247418 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247424 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:10:47.250023 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247426 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247429 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247432 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247435 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247438 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247441 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247443 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247446 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247449 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247452 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247455 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247459 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247465 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247469 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247471 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247474 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247477 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247480 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247483 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247486 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:10:47.250535 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247489 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247492 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247495 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247498 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247503 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247514 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247517 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247520 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247523 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247526 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247528 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247531 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247534 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247537 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247542 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247546 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247551 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247555 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247557 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247561 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:10:47.251084 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247564 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247567 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247570 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247573 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247578 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247582 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247585 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247588 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247594 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247597 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247600 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247603 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247606 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247612 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247615 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247619 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247622 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247625 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247634 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:10:47.251617 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247637 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247643 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247646 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247649 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247652 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247673 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247677 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247680 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247683 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247686 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247690 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247693 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247696 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247702 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247705 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247708 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247712 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.247715 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248024 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248034 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248043 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 19:10:47.252128 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248048 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248053 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248056 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248069 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248074 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248077 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248081 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248084 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248088 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248091 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248094 2576 flags.go:64] FLAG: --cgroup-root="" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248098 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248102 2576 flags.go:64] FLAG: --client-ca-file="" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248105 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248108 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248113 2576 flags.go:64] FLAG: --cluster-domain="" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248119 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248122 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248126 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248129 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248133 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248137 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248140 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 19:10:47.252696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248144 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248147 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248150 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248154 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248158 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248161 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248166 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248169 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248172 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248176 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248179 2576 flags.go:64] FLAG: --enable-server="true" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248184 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248191 2576 flags.go:64] FLAG: --event-burst="100" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248195 2576 flags.go:64] FLAG: --event-qps="50" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248199 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248203 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248208 2576 flags.go:64] FLAG: --eviction-hard="" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248214 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248218 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248222 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248231 2576 flags.go:64] FLAG: --eviction-soft="" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248234 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248240 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248243 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248248 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 19:10:47.253310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248253 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248256 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248259 2576 flags.go:64] FLAG: --feature-gates="" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248263 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248266 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248270 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248273 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248276 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248280 2576 flags.go:64] FLAG: --help="false" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248283 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-128-74.ec2.internal" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248286 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248290 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248293 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248296 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248300 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248303 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248307 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248310 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248313 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248317 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248320 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248323 2576 flags.go:64] FLAG: --kube-reserved="" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248326 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248329 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 19:10:47.253977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248332 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248335 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248338 2576 flags.go:64] FLAG: --lock-file="" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248343 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248347 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248350 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248356 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248367 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248372 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248375 2576 flags.go:64] FLAG: --logging-format="text" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248378 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248382 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248386 2576 flags.go:64] FLAG: --manifest-url="" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248389 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248399 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248403 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248407 2576 flags.go:64] FLAG: --max-pods="110" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248411 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248414 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248417 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248420 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248423 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248426 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248429 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248439 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 19:10:47.254737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248442 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248446 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248449 2576 flags.go:64] FLAG: --pod-cidr="" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248452 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248460 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248463 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248466 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248469 2576 flags.go:64] FLAG: --port="10250" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248473 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248476 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-045bc24f842669b0d" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248481 2576 flags.go:64] FLAG: --qos-reserved="" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248484 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248487 2576 flags.go:64] FLAG: --register-node="true" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248490 2576 flags.go:64] FLAG: --register-with-taints="" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248495 2576 flags.go:64] FLAG: --registry-burst="10" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248506 2576 flags.go:64] FLAG: --registry-qps="5" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248511 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248514 2576 flags.go:64] FLAG: --reserved-memory="" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248518 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248522 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248525 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248528 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248531 2576 flags.go:64] FLAG: --runonce="false" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248534 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248538 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 19:10:47.255424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248541 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248544 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248547 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248551 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248554 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248558 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248560 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248564 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248567 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248570 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248573 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248576 2576 flags.go:64] FLAG: --system-cgroups="" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248579 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248586 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248589 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248592 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248599 2576 flags.go:64] FLAG: --tls-min-version="" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248605 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248608 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248611 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248614 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248618 2576 flags.go:64] FLAG: --v="2" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248622 2576 flags.go:64] FLAG: --version="false" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248634 2576 flags.go:64] FLAG: --vmodule="" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248639 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 19:10:47.256152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.248643 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248796 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248801 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248805 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248808 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248811 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248816 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248819 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248823 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248826 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248829 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248832 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248835 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248838 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248840 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248843 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248846 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248849 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248851 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248854 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:10:47.256943 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248857 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248860 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248863 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248865 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248870 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248873 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248876 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248878 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248882 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248884 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248889 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248899 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248902 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248905 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248908 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248911 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248914 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248916 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248919 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248922 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:10:47.257491 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248925 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248928 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248930 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248933 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248936 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248939 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248941 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248944 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248947 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248950 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248952 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248956 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248959 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248961 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248964 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248967 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248971 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248975 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248979 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248982 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:10:47.258033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248985 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248989 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248994 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.248998 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249008 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249011 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249014 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249017 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249020 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249023 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249025 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249029 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249032 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249034 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249037 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249040 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249042 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249045 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249048 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:10:47.258570 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249051 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249054 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249056 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249059 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249062 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249064 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249067 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249070 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249072 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249077 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249080 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249083 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249085 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249089 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249093 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249096 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249099 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249102 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249113 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:10:47.259081 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249116 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:10:47.259582 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.249119 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:10:47.259582 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.250131 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 19:10:47.259582 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.258952 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 19:10:47.259582 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.258972 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 19:10:47.259979 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.259967 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 19:10:47.260013 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.259980 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 19:10:47.260064 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260054 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:10:47.260064 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260063 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260067 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260071 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260074 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260078 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260081 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260084 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260088 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260093 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260096 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260100 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260103 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260106 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260109 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260112 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260115 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260118 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260121 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260123 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260126 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:10:47.260122 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260130 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260133 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260136 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260140 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260142 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260145 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260148 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260151 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260155 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260158 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260161 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260163 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260168 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260172 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260175 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260178 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260181 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260184 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260187 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:10:47.260644 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260189 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260192 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260195 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260198 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260201 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260204 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260207 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260210 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260212 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260215 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260218 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260221 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260223 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260226 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260229 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260232 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260236 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260239 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260242 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260246 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:10:47.261162 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260248 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260252 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260256 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260259 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260261 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260264 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260267 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260270 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260273 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260275 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260278 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260281 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260284 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260287 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260290 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260293 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260296 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260298 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260301 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260305 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:10:47.261703 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260309 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:10:47.262230 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260312 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:10:47.262230 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260315 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:10:47.262230 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260318 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:10:47.262230 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260321 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:10:47.262230 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260323 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:10:47.262230 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260326 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:10:47.262230 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260329 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:10:47.262230 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260332 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:10:47.262230 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260336 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:10:47.262230 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260339 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:10:47.262230 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260343 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:10:47.262230 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260346 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:10:47.262230 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260349 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:10:47.262230 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260352 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:10:47.262230 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260356 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:10:47.262230 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260358 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:10:47.262230 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260361 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:10:47.262230 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260364 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:10:47.262712 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.260370 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 19:10:47.262712 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260521 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:10:47.262712 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260527 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:10:47.262712 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260530 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:10:47.262712 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260534 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:10:47.262712 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260537 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:10:47.262712 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260540 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:10:47.262712 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260543 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:10:47.262712 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260546 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:10:47.262712 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260549 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:10:47.262712 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260552 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:10:47.262712 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260555 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:10:47.262712 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260558 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:10:47.262712 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260561 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:10:47.262712 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260564 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260567 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260570 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260573 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260577 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260579 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260582 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260585 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260588 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260590 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260593 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260596 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260599 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260602 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260605 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260608 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260611 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260614 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260617 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:10:47.263108 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260620 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260623 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260626 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260629 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260645 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260648 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260651 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260654 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260660 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260662 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260665 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260669 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260671 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260675 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260678 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260681 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260684 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260686 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260689 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260692 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:10:47.263652 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260695 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260698 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260701 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260704 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260706 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260709 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260712 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260715 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260718 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260722 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260725 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260728 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260731 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260733 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260736 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260739 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260743 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260746 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260750 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260752 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:10:47.264262 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260755 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260758 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260761 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260764 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260767 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260770 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260773 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260777 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260781 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260784 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260787 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260790 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260793 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260795 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260798 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260810 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260814 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260816 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260819 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260822 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:10:47.264811 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260825 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:10:47.265445 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260827 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:10:47.265445 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260831 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:10:47.265445 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260833 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:10:47.265445 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260836 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:10:47.265445 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260840 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:10:47.265445 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:47.260844 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:10:47.265445 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.260850 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 19:10:47.265445 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.260989 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 19:10:47.265445 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.263326 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 19:10:47.265445 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.264723 2576 server.go:1019] "Starting client certificate rotation" Jul 14 19:10:47.265445 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.264855 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 19:10:47.265445 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.264913 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 19:10:47.267742 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.267723 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 19:10:47.269416 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.269390 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 19:10:47.285520 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.285495 2576 log.go:25] "Validated CRI v1 runtime API" Jul 14 19:10:47.287762 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.287744 2576 log.go:25] "Validated CRI v1 image API" Jul 14 19:10:47.288622 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.288605 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 19:10:47.291473 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.291447 2576 fs.go:135] Filesystem UUIDs: map[416f14ac-7598-4f31-8499-eb7ad4be2f23:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ea7c5269-6167-4ba0-b732-f55ef849cc76:/dev/nvme0n1p4] Jul 14 19:10:47.291526 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.291474 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 19:10:47.299499 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.299345 2576 manager.go:216] Machine: {Timestamp:2026-07-14 19:10:47.298746188 +0000 UTC m=+0.411996052 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097309 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aa83a4c6db2683813a4322e9358d4 SystemUUID:ec2aa83a-4c6d-b268-3813-a4322e9358d4 BootID:957abbc3-56fe-4f65-98c8-69bd26bedfb5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:12:e8:ae:e8:4f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:12:e8:ae:e8:4f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:45:28:9e:6b:33 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 19:10:47.299499 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.299476 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 19:10:47.299686 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.299585 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 19:10:47.303723 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.303681 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 19:10:47.303874 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.303728 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-74.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 19:10:47.304505 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.304488 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 19:10:47.304644 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.304631 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 19:10:47.304702 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.304649 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 19:10:47.304702 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.304673 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 19:10:47.304702 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.304687 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 19:10:47.305645 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.305634 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 19:10:47.309431 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.309419 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 14 19:10:47.309561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.309552 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 19:10:47.309597 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.309585 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 14 19:10:47.309626 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.309598 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 19:10:47.309626 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.309615 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 19:10:47.309626 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.309625 2576 kubelet.go:398] "Adding apiserver pod source" Jul 14 19:10:47.309752 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.309634 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 19:10:47.310077 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.310064 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 19:10:47.310137 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.310084 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 19:10:47.312845 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.312824 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 19:10:47.314533 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.314514 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 19:10:47.314634 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.314556 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 19:10:47.315416 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.315401 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 19:10:47.315480 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.315421 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 19:10:47.315480 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.315431 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 19:10:47.315480 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.315439 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 19:10:47.315480 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.315452 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 19:10:47.315480 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.315462 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 19:10:47.315700 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.315542 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 19:10:47.315700 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.315552 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 19:10:47.315700 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.315563 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 19:10:47.315700 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.315572 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 19:10:47.315700 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.315587 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 19:10:47.315700 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.315601 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 19:10:47.317691 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.317653 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 19:10:47.317753 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.317693 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 19:10:47.321374 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.321356 2576 server.go:1268] "Started kubelet" Jul 14 19:10:47.321508 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.321460 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 19:10:47.321808 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.321675 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 19:10:47.321903 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.321864 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 19:10:47.322415 ip-10-0-128-74 systemd[1]: Started Kubernetes Kubelet. Jul 14 19:10:47.322664 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.322613 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 19:10:47.322737 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.322641 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 19:10:47.322799 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.322773 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-74.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 19:10:47.323583 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.323566 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 19:10:47.324610 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.324583 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 14 19:10:47.328123 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.328103 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 19:10:47.329856 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.329832 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 19:10:47.329856 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.329840 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 19:10:47.330053 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.329984 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-74.ec2.internal\" not found" Jul 14 19:10:47.331977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.330692 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 19:10:47.331977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.330695 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 19:10:47.331977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.330864 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 19:10:47.331977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.331005 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 14 19:10:47.331977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.331016 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 14 19:10:47.331977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.331144 2576 factory.go:55] Registering systemd factory Jul 14 19:10:47.331977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.331158 2576 factory.go:223] Registration of the systemd container factory successfully Jul 14 19:10:47.331977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.331662 2576 factory.go:153] Registering CRI-O factory Jul 14 19:10:47.331977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.331680 2576 factory.go:223] Registration of the crio container factory successfully Jul 14 19:10:47.331977 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.331882 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 14 19:10:47.332778 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.331474 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-74.ec2.internal.18c23db460e77924 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-74.ec2.internal,UID:ip-10-0-128-74.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-74.ec2.internal,},FirstTimestamp:2026-07-14 19:10:47.321327908 +0000 UTC m=+0.434577773,LastTimestamp:2026-07-14 19:10:47.321327908 +0000 UTC m=+0.434577773,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-74.ec2.internal,}" Jul 14 19:10:47.333749 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.333730 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 19:10:47.333844 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.333768 2576 factory.go:103] Registering Raw factory Jul 14 19:10:47.333844 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.333788 2576 manager.go:1196] Started watching for new ooms in manager Jul 14 19:10:47.335109 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.335088 2576 manager.go:349] Starting recovery of all containers Jul 14 19:10:47.338860 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.338812 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 14 19:10:47.346864 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.346739 2576 manager.go:354] Recovery completed Jul 14 19:10:47.351079 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.351064 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:10:47.353218 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.353188 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:10:47.353300 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.353235 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:10:47.353300 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.353250 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:10:47.353837 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.353817 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 19:10:47.353837 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.353834 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 19:10:47.353990 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.353855 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 14 19:10:47.353990 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.353833 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:10:47.355127 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.355107 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:10:47.355209 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.355136 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:10:47.355209 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.355149 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:10:47.355297 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.355251 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8csxr" Jul 14 19:10:47.355342 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.355328 2576 policy_none.go:49] "None policy: Start" Jul 14 19:10:47.355373 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.355349 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 19:10:47.355402 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.355380 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 19:10:47.356105 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.356030 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-74.ec2.internal.18c23db462ce1211 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-74.ec2.internal,UID:ip-10-0-128-74.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-74.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-74.ec2.internal,},FirstTimestamp:2026-07-14 19:10:47.353217553 +0000 UTC m=+0.466467416,LastTimestamp:2026-07-14 19:10:47.353217553 +0000 UTC m=+0.466467416,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-74.ec2.internal,}" Jul 14 19:10:47.356689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.356679 2576 policy_none.go:47] "Start" Jul 14 19:10:47.364130 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.364112 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8csxr" Jul 14 19:10:47.368400 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.368260 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-74.ec2.internal.18c23db462ce72c7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-74.ec2.internal,UID:ip-10-0-128-74.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-74.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-74.ec2.internal,},FirstTimestamp:2026-07-14 19:10:47.353242311 +0000 UTC m=+0.466492175,LastTimestamp:2026-07-14 19:10:47.353242311 +0000 UTC m=+0.466492175,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-74.ec2.internal,}" Jul 14 19:10:47.407400 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.407378 2576 manager.go:337] "Starting Device Plugin manager" Jul 14 19:10:47.407540 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.407428 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 19:10:47.407540 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.407443 2576 server.go:87] "Starting device plugin registration server" Jul 14 19:10:47.407803 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.407790 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 19:10:47.407865 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.407805 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 19:10:47.407946 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.407924 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 19:10:47.408615 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.408579 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 19:10:47.408615 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.408611 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 19:10:47.408794 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.408694 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 19:10:47.408794 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.408759 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-74.ec2.internal\" not found" Jul 14 19:10:47.507173 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.507076 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 19:10:47.508471 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.508357 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:10:47.508471 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.508421 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 19:10:47.508471 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.508460 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 19:10:47.508675 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.508489 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 19:10:47.508675 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.508535 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 19:10:47.509931 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.509906 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:10:47.510039 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.509953 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:10:47.510039 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.509970 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:10:47.510039 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.509999 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-74.ec2.internal" Jul 14 19:10:47.511044 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.511025 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:10:47.519939 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.519913 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-74.ec2.internal" Jul 14 19:10:47.519939 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.519943 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-74.ec2.internal\": node \"ip-10-0-128-74.ec2.internal\" not found" Jul 14 19:10:47.548104 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.548069 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-74.ec2.internal\" not found" Jul 14 19:10:47.610673 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.610622 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-74.ec2.internal"] Jul 14 19:10:47.610772 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.610730 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:10:47.611780 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.611761 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:10:47.611877 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.611793 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:10:47.611877 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.611812 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:10:47.613377 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.613363 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:10:47.613524 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.613508 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal" Jul 14 19:10:47.613562 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.613544 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:10:47.614222 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.614203 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:10:47.614297 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.614236 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:10:47.614297 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.614248 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:10:47.614361 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.614203 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:10:47.614361 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.614314 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:10:47.614361 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.614328 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:10:47.615622 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.615608 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-74.ec2.internal" Jul 14 19:10:47.615702 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.615633 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:10:47.616400 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.616378 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:10:47.616490 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.616403 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:10:47.616490 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.616417 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:10:47.630335 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.630304 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-74.ec2.internal\" not found" node="ip-10-0-128-74.ec2.internal" Jul 14 19:10:47.632025 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.631996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc14085ea90483c50f2a94112a007054-config\") pod \"kube-apiserver-proxy-ip-10-0-128-74.ec2.internal\" (UID: \"cc14085ea90483c50f2a94112a007054\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-74.ec2.internal" Jul 14 19:10:47.632157 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.632033 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/471487e6885ab16f5c27e006ead6368f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal\" (UID: \"471487e6885ab16f5c27e006ead6368f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal" Jul 14 19:10:47.632157 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.632060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/471487e6885ab16f5c27e006ead6368f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal\" (UID: \"471487e6885ab16f5c27e006ead6368f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal" Jul 14 19:10:47.634204 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.634183 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-74.ec2.internal\" not found" node="ip-10-0-128-74.ec2.internal" Jul 14 19:10:47.648258 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.648232 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-74.ec2.internal\" not found" Jul 14 19:10:47.732745 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.732717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/471487e6885ab16f5c27e006ead6368f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal\" (UID: \"471487e6885ab16f5c27e006ead6368f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal" Jul 14 19:10:47.732892 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.732783 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/471487e6885ab16f5c27e006ead6368f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal\" (UID: \"471487e6885ab16f5c27e006ead6368f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal" Jul 14 19:10:47.732892 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.732812 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/471487e6885ab16f5c27e006ead6368f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal\" (UID: \"471487e6885ab16f5c27e006ead6368f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal" Jul 14 19:10:47.732892 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.732839 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/471487e6885ab16f5c27e006ead6368f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal\" (UID: \"471487e6885ab16f5c27e006ead6368f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal" Jul 14 19:10:47.733011 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.732857 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc14085ea90483c50f2a94112a007054-config\") pod \"kube-apiserver-proxy-ip-10-0-128-74.ec2.internal\" (UID: \"cc14085ea90483c50f2a94112a007054\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-74.ec2.internal" Jul 14 19:10:47.733011 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.732946 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc14085ea90483c50f2a94112a007054-config\") pod \"kube-apiserver-proxy-ip-10-0-128-74.ec2.internal\" (UID: \"cc14085ea90483c50f2a94112a007054\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-74.ec2.internal" Jul 14 19:10:47.749051 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.749020 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-74.ec2.internal\" not found" Jul 14 19:10:47.849436 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.849342 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-74.ec2.internal\" not found" Jul 14 19:10:47.936223 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.936182 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal" Jul 14 19:10:47.937981 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:47.937951 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-74.ec2.internal" Jul 14 19:10:47.949840 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:47.949815 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-74.ec2.internal\" not found" Jul 14 19:10:48.050117 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:48.050073 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-74.ec2.internal\" not found" Jul 14 19:10:48.151153 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:48.151073 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-74.ec2.internal\" not found" Jul 14 19:10:48.251206 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:48.251176 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-74.ec2.internal\" not found" Jul 14 19:10:48.264671 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:48.264641 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 19:10:48.264825 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:48.264806 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 19:10:48.351320 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:48.351283 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-74.ec2.internal\" not found" Jul 14 19:10:48.369420 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:48.369369 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 19:05:47 +0000 UTC" deadline="2028-02-11 12:13:00.21233695 +0000 UTC" Jul 14 19:10:48.369420 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:48.369412 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13841h2m11.842928439s" Jul 14 19:10:48.451682 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:48.451556 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-74.ec2.internal\" not found" Jul 14 19:10:48.541909 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:48.541874 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:10:48.543615 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:48.543594 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:10:48.543765 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:48.543629 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:10:48.543765 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:48.543650 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:10:48.552121 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:48.552100 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-74.ec2.internal\" not found" Jul 14 19:10:48.653190 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:48.653150 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-74.ec2.internal\" not found" Jul 14 19:10:48.665436 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:48.665392 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc14085ea90483c50f2a94112a007054.slice/crio-2cf42cb6b8d42a5b4c6ed0878170162235f18c3bdec62266da0429cdeb0ac358 WatchSource:0}: Error finding container 2cf42cb6b8d42a5b4c6ed0878170162235f18c3bdec62266da0429cdeb0ac358: Status 404 returned error can't find the container with id 2cf42cb6b8d42a5b4c6ed0878170162235f18c3bdec62266da0429cdeb0ac358 Jul 14 19:10:48.671923 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:48.671905 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:10:48.723381 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:48.723283 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:10:48.734199 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:48.734175 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:10:48.754143 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:48.754103 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-74.ec2.internal\" not found" Jul 14 19:10:48.855275 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:48.855234 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-74.ec2.internal\" not found" Jul 14 19:10:48.898563 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:48.898532 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:10:48.932455 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:48.932425 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-74.ec2.internal" Jul 14 19:10:48.944227 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:48.944197 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 19:10:48.945911 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:48.945897 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal" Jul 14 19:10:48.959424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:48.959404 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 19:10:49.312925 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.312883 2576 apiserver.go:52] "Watching apiserver" Jul 14 19:10:49.322027 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.321996 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 19:10:49.323251 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.323225 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-8k4j8","openshift-image-registry/node-ca-z4lnk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal","openshift-multus/multus-additional-cni-plugins-f7zqk","openshift-multus/multus-pxrgt","openshift-multus/network-metrics-daemon-xczln","kube-system/global-pull-secret-syncer-bv2wz","kube-system/konnectivity-agent-bb6wd","openshift-dns/node-resolver-q6rws","openshift-network-diagnostics/network-check-target-tgldx","openshift-network-operator/iptables-alerter-msbnp","openshift-ovn-kubernetes/ovnkube-node-tg2x9","kube-system/kube-apiserver-proxy-ip-10-0-128-74.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct"] Jul 14 19:10:49.325990 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.325963 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:10:49.326153 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:49.326050 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:10:49.327394 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.327374 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.328743 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.328724 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q6rws" Jul 14 19:10:49.330121 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.330105 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bb6wd" Jul 14 19:10:49.330243 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.330204 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 19:10:49.331495 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.331478 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.332852 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.332835 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:10:49.332955 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:49.332918 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:10:49.334206 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.334188 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.335524 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.335508 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.336153 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.336133 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 19:10:49.336253 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.336137 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 19:10:49.336972 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.336953 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:10:49.337074 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:49.337032 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:10:49.337403 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.337382 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 19:10:49.337403 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.337391 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 19:10:49.337604 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.337421 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 19:10:49.337604 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.337460 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4zgtr\"" Jul 14 19:10:49.337604 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.337466 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 19:10:49.337887 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.337798 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j2b4k\"" Jul 14 19:10:49.337887 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.337848 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 19:10:49.337887 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.337857 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:10:49.338074 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.337991 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 19:10:49.338617 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.338579 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-msbnp" Jul 14 19:10:49.340010 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.339980 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 19:10:49.340283 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.340266 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 19:10:49.340396 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.340386 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 19:10:49.340396 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.340390 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f2wdd\"" Jul 14 19:10:49.340573 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.340391 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 19:10:49.340573 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.340560 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z4lnk" Jul 14 19:10:49.340745 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.340573 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 19:10:49.340745 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.340564 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 19:10:49.340745 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.340725 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 19:10:49.341413 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.341398 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-45f4p\"" Jul 14 19:10:49.341413 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.341407 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 19:10:49.341571 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.341420 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-svlgl\"" Jul 14 19:10:49.341571 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.341475 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mh4bp\"" Jul 14 19:10:49.341571 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.341420 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 19:10:49.341811 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.341790 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2vrb6\"" Jul 14 19:10:49.341924 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.341796 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 19:10:49.342145 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.342124 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 19:10:49.342217 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.342124 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:10:49.342285 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.342258 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.343454 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-hostroot\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.343553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343461 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-multus-conf-dir\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.343553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343479 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-kubelet\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.343553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-host-var-lib-cni-bin\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.343553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e563268e-31ea-4ddb-82ff-da5017d5a3fb-multus-daemon-config\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.343888 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343555 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-host-run-multus-certs\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.343888 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343603 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-etc-kubernetes\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.343888 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343647 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh2xz\" (UniqueName: \"kubernetes.io/projected/e563268e-31ea-4ddb-82ff-da5017d5a3fb-kube-api-access-zh2xz\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.343888 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343671 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghgdc\" (UniqueName: \"kubernetes.io/projected/8d1706ff-a844-42e9-b6d8-628495d88055-kube-api-access-ghgdc\") pod \"network-metrics-daemon-xczln\" (UID: \"8d1706ff-a844-42e9-b6d8-628495d88055\") " pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:10:49.343888 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343692 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-run-openvswitch\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.343888 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-cni-bin\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.343888 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343741 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-ovn-node-metrics-cert\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.343888 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343763 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-run-ovn\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.343888 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-ovnkube-config\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.343888 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343800 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-ovnkube-script-lib\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.343888 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343821 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/659ae1d5-f1bd-4bec-b2c9-4b5fe3ae0a8d-konnectivity-ca\") pod \"konnectivity-agent-bb6wd\" (UID: \"659ae1d5-f1bd-4bec-b2c9-4b5fe3ae0a8d\") " pod="kube-system/konnectivity-agent-bb6wd" Jul 14 19:10:49.343888 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343836 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-os-release\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.343888 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-host-var-lib-cni-multus\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.343888 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-etc-openvswitch\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.343888 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343899 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-run-ovn-kubernetes\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.344561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343926 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a0b990-bc45-4764-bd78-31069184c58a-tmp-dir\") pod \"node-resolver-q6rws\" (UID: \"e0a0b990-bc45-4764-bd78-31069184c58a\") " pod="openshift-dns/node-resolver-q6rws" Jul 14 19:10:49.344561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343942 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p6kq\" (UniqueName: \"kubernetes.io/projected/e0a0b990-bc45-4764-bd78-31069184c58a-kube-api-access-4p6kq\") pod \"node-resolver-q6rws\" (UID: \"e0a0b990-bc45-4764-bd78-31069184c58a\") " pod="openshift-dns/node-resolver-q6rws" Jul 14 19:10:49.344561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.343989 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-system-cni-dir\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.344561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344029 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e563268e-31ea-4ddb-82ff-da5017d5a3fb-cni-binary-copy\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.344561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344064 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-systemd-units\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.344561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344095 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.344561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344120 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-env-overrides\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.344561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344141 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/659ae1d5-f1bd-4bec-b2c9-4b5fe3ae0a8d-agent-certs\") pod \"konnectivity-agent-bb6wd\" (UID: \"659ae1d5-f1bd-4bec-b2c9-4b5fe3ae0a8d\") " pod="kube-system/konnectivity-agent-bb6wd" Jul 14 19:10:49.344561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344196 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-multus-cni-dir\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.344561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344226 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-cnibin\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.344561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344253 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-host-var-lib-kubelet\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.344561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344280 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-slash\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.344561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-host-run-netns\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.344561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344329 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs\") pod \"network-metrics-daemon-xczln\" (UID: \"8d1706ff-a844-42e9-b6d8-628495d88055\") " pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:10:49.344561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344376 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-run-netns\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.344561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344410 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-run-systemd\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.344561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k676k\" (UniqueName: \"kubernetes.io/projected/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-kube-api-access-k676k\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.345174 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344465 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-multus-socket-dir-parent\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.345174 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344503 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-var-lib-openvswitch\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.345174 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344537 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-node-log\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.345174 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344567 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-log-socket\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.345174 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344625 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-cni-netd\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.345174 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0a0b990-bc45-4764-bd78-31069184c58a-hosts-file\") pod \"node-resolver-q6rws\" (UID: \"e0a0b990-bc45-4764-bd78-31069184c58a\") " pod="openshift-dns/node-resolver-q6rws" Jul 14 19:10:49.345174 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.344677 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-host-run-k8s-cni-cncf-io\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.352342 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.352312 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 19:10:49.352342 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.352329 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 19:10:49.352529 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.352357 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 19:10:49.352529 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.352371 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 19:10:49.352529 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.352312 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 19:10:49.352529 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.352317 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fgkbc\"" Jul 14 19:10:49.352529 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.352399 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 19:10:49.352893 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.352876 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l9hsf\"" Jul 14 19:10:49.361407 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.361384 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 19:10:49.379634 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.379611 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x5r4s" Jul 14 19:10:49.388454 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.388431 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x5r4s" Jul 14 19:10:49.431658 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.431635 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 19:10:49.445109 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-cni-bin\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.445206 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445120 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-ovn-node-metrics-cert\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.445206 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445141 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlf7j\" (UniqueName: \"kubernetes.io/projected/5dc12361-b85d-43a1-a491-e4072a9336b1-kube-api-access-tlf7j\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.445206 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445166 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-cni-bin\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.445206 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445191 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70375e8e-ae6d-4ce1-8240-012361054757-serviceca\") pod \"node-ca-z4lnk\" (UID: \"70375e8e-ae6d-4ce1-8240-012361054757\") " pod="openshift-image-registry/node-ca-z4lnk" Jul 14 19:10:49.445333 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445230 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e6c415f-305f-477e-b939-1c2b84a3d770-dbus\") pod \"global-pull-secret-syncer-bv2wz\" (UID: \"5e6c415f-305f-477e-b939-1c2b84a3d770\") " pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:10:49.445333 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-run-ovn\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.445333 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-ovnkube-config\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.445333 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445331 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/659ae1d5-f1bd-4bec-b2c9-4b5fe3ae0a8d-konnectivity-ca\") pod \"konnectivity-agent-bb6wd\" (UID: \"659ae1d5-f1bd-4bec-b2c9-4b5fe3ae0a8d\") " pod="kube-system/konnectivity-agent-bb6wd" Jul 14 19:10:49.445582 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-os-release\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.445582 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-lib-modules\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.445582 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-etc-openvswitch\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.445582 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445485 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-os-release\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.445582 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-run-ovn-kubernetes\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.445582 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445502 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-run-ovn\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.445582 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445544 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-etc-openvswitch\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.445980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445598 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 19:10:49.445980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445620 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-run-ovn-kubernetes\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.445980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a0b990-bc45-4764-bd78-31069184c58a-tmp-dir\") pod \"node-resolver-q6rws\" (UID: \"e0a0b990-bc45-4764-bd78-31069184c58a\") " pod="openshift-dns/node-resolver-q6rws" Jul 14 19:10:49.445980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-system-cni-dir\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.445980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445694 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-modprobe-d\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.445980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-sys\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.445980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445733 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-host\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.445980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dc12361-b85d-43a1-a491-e4072a9336b1-tmp\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.445980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-systemd-units\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.445980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445798 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-env-overrides\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.445980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445826 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-cnibin\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.445980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-host-var-lib-kubelet\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.445980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-systemd\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.445980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445931 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-ovnkube-config\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.445980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445981 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-system-cni-dir\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.445980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445987 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-host-var-lib-kubelet\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.446689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445997 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70375e8e-ae6d-4ce1-8240-012361054757-host\") pod \"node-ca-z4lnk\" (UID: \"70375e8e-ae6d-4ce1-8240-012361054757\") " pod="openshift-image-registry/node-ca-z4lnk" Jul 14 19:10:49.446689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.445998 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-systemd-units\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.446689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446037 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-cnibin\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.446689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446081 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gx7v\" (UniqueName: \"kubernetes.io/projected/81bf32b9-26cc-4396-8d68-9fc1c306e620-kube-api-access-7gx7v\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.446689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446079 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/659ae1d5-f1bd-4bec-b2c9-4b5fe3ae0a8d-konnectivity-ca\") pod \"konnectivity-agent-bb6wd\" (UID: \"659ae1d5-f1bd-4bec-b2c9-4b5fe3ae0a8d\") " pod="kube-system/konnectivity-agent-bb6wd" Jul 14 19:10:49.446689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-run-netns\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.446689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446165 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-kubernetes\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.446689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446172 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-run-netns\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.446689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446210 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-tuned\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.446689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446239 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d475a4f-8acf-4cc1-b605-be9c1be74d34-cnibin\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.446689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446265 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-node-log\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.446689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-cni-netd\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.446689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446329 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-env-overrides\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.446689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446338 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-node-log\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.446689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446378 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0a0b990-bc45-4764-bd78-31069184c58a-hosts-file\") pod \"node-resolver-q6rws\" (UID: \"e0a0b990-bc45-4764-bd78-31069184c58a\") " pod="openshift-dns/node-resolver-q6rws" Jul 14 19:10:49.446689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446394 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-cni-netd\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.446689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-multus-conf-dir\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.446689 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446452 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0a0b990-bc45-4764-bd78-31069184c58a-hosts-file\") pod \"node-resolver-q6rws\" (UID: \"e0a0b990-bc45-4764-bd78-31069184c58a\") " pod="openshift-dns/node-resolver-q6rws" Jul 14 19:10:49.447553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-sysctl-d\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.447553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446503 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-multus-conf-dir\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.447553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446523 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-host-var-lib-cni-bin\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.447553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446562 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-host-var-lib-cni-bin\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.447553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446576 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e563268e-31ea-4ddb-82ff-da5017d5a3fb-multus-daemon-config\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.447553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446602 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a0b990-bc45-4764-bd78-31069184c58a-tmp-dir\") pod \"node-resolver-q6rws\" (UID: \"e0a0b990-bc45-4764-bd78-31069184c58a\") " pod="openshift-dns/node-resolver-q6rws" Jul 14 19:10:49.447553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh2xz\" (UniqueName: \"kubernetes.io/projected/e563268e-31ea-4ddb-82ff-da5017d5a3fb-kube-api-access-zh2xz\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.447553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446645 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghgdc\" (UniqueName: \"kubernetes.io/projected/8d1706ff-a844-42e9-b6d8-628495d88055-kube-api-access-ghgdc\") pod \"network-metrics-daemon-xczln\" (UID: \"8d1706ff-a844-42e9-b6d8-628495d88055\") " pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:10:49.447553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446664 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x7rx\" (UniqueName: \"kubernetes.io/projected/ec17ab27-ef49-4fae-9006-b236a56a1727-kube-api-access-8x7rx\") pod \"iptables-alerter-msbnp\" (UID: \"ec17ab27-ef49-4fae-9006-b236a56a1727\") " pod="openshift-network-operator/iptables-alerter-msbnp" Jul 14 19:10:49.447553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/81bf32b9-26cc-4396-8d68-9fc1c306e620-etc-selinux\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.447553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-run-openvswitch\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.447553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446766 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e6c415f-305f-477e-b939-1c2b84a3d770-kubelet-config\") pod \"global-pull-secret-syncer-bv2wz\" (UID: \"5e6c415f-305f-477e-b939-1c2b84a3d770\") " pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:10:49.447553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446793 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-ovnkube-script-lib\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.447553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446805 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-run-openvswitch\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.447553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446841 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-host-var-lib-cni-multus\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.447553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446866 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-var-lib-kubelet\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.447553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446882 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d475a4f-8acf-4cc1-b605-be9c1be74d34-os-release\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.448327 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446931 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-host-var-lib-cni-multus\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.448327 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.446921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d475a4f-8acf-4cc1-b605-be9c1be74d34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.448327 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447002 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret\") pod \"global-pull-secret-syncer-bv2wz\" (UID: \"5e6c415f-305f-477e-b939-1c2b84a3d770\") " pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:10:49.448327 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p6kq\" (UniqueName: \"kubernetes.io/projected/e0a0b990-bc45-4764-bd78-31069184c58a-kube-api-access-4p6kq\") pod \"node-resolver-q6rws\" (UID: \"e0a0b990-bc45-4764-bd78-31069184c58a\") " pod="openshift-dns/node-resolver-q6rws" Jul 14 19:10:49.448327 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447060 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e563268e-31ea-4ddb-82ff-da5017d5a3fb-cni-binary-copy\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.448327 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447082 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d475a4f-8acf-4cc1-b605-be9c1be74d34-system-cni-dir\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.448327 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447128 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec17ab27-ef49-4fae-9006-b236a56a1727-iptables-alerter-script\") pod \"iptables-alerter-msbnp\" (UID: \"ec17ab27-ef49-4fae-9006-b236a56a1727\") " pod="openshift-network-operator/iptables-alerter-msbnp" Jul 14 19:10:49.448327 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447136 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e563268e-31ea-4ddb-82ff-da5017d5a3fb-multus-daemon-config\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.448327 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447170 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/81bf32b9-26cc-4396-8d68-9fc1c306e620-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.448327 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447199 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.448327 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447226 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/659ae1d5-f1bd-4bec-b2c9-4b5fe3ae0a8d-agent-certs\") pod \"konnectivity-agent-bb6wd\" (UID: \"659ae1d5-f1bd-4bec-b2c9-4b5fe3ae0a8d\") " pod="kube-system/konnectivity-agent-bb6wd" Jul 14 19:10:49.448327 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447291 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-multus-cni-dir\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.448327 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-sysctl-conf\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.448327 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-run\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.448327 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447361 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-ovnkube-script-lib\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.448327 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447382 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d475a4f-8acf-4cc1-b605-be9c1be74d34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.448327 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447389 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-multus-cni-dir\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.448870 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447323 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.448870 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwg6s\" (UniqueName: \"kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s\") pod \"network-check-target-tgldx\" (UID: \"67bb23c7-9bee-4350-a4bf-0da1589fbce0\") " pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:10:49.448870 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-slash\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.448870 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447471 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-host-run-netns\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.448870 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447490 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs\") pod \"network-metrics-daemon-xczln\" (UID: \"8d1706ff-a844-42e9-b6d8-628495d88055\") " pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:10:49.448870 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d475a4f-8acf-4cc1-b605-be9c1be74d34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.448870 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447539 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-host-run-netns\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.448870 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec17ab27-ef49-4fae-9006-b236a56a1727-host-slash\") pod \"iptables-alerter-msbnp\" (UID: \"ec17ab27-ef49-4fae-9006-b236a56a1727\") " pod="openshift-network-operator/iptables-alerter-msbnp" Jul 14 19:10:49.448870 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447520 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-slash\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.448870 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:49.447617 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:10:49.448870 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fcgx\" (UniqueName: \"kubernetes.io/projected/70375e8e-ae6d-4ce1-8240-012361054757-kube-api-access-8fcgx\") pod \"node-ca-z4lnk\" (UID: \"70375e8e-ae6d-4ce1-8240-012361054757\") " pod="openshift-image-registry/node-ca-z4lnk" Jul 14 19:10:49.448870 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447653 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e563268e-31ea-4ddb-82ff-da5017d5a3fb-cni-binary-copy\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.448870 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/81bf32b9-26cc-4396-8d68-9fc1c306e620-sys-fs\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.448870 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:49.447721 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs podName:8d1706ff-a844-42e9-b6d8-628495d88055 nodeName:}" failed. No retries permitted until 2026-07-14 19:10:49.947676162 +0000 UTC m=+3.060926033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs") pod "network-metrics-daemon-xczln" (UID: "8d1706ff-a844-42e9-b6d8-628495d88055") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:10:49.448870 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-run-systemd\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.448870 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k676k\" (UniqueName: \"kubernetes.io/projected/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-kube-api-access-k676k\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.448870 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-multus-socket-dir-parent\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.449393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-var-lib-openvswitch\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.449393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-log-socket\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.449393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447918 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-log-socket\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.449393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447924 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-var-lib-openvswitch\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.449393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447957 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-run-systemd\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.449393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-host-run-k8s-cni-cncf-io\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.449393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447983 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-multus-socket-dir-parent\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.449393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.447996 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-host-run-k8s-cni-cncf-io\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.449393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.448007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-hostroot\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.449393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.448036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-sysconfig\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.449393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.448039 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-hostroot\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.449393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.448060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d475a4f-8acf-4cc1-b605-be9c1be74d34-cni-binary-copy\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.449393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.448085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/81bf32b9-26cc-4396-8d68-9fc1c306e620-socket-dir\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.449393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.448107 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/81bf32b9-26cc-4396-8d68-9fc1c306e620-registration-dir\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.449393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.448136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-kubelet\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.449393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.448161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-host-run-multus-certs\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.449393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.448187 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-etc-kubernetes\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.449943 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.448218 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-host-kubelet\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.449943 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.448216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2hfg\" (UniqueName: \"kubernetes.io/projected/2d475a4f-8acf-4cc1-b605-be9c1be74d34-kube-api-access-h2hfg\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.449943 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.448233 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-host-run-multus-certs\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.449943 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.448261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/81bf32b9-26cc-4396-8d68-9fc1c306e620-device-dir\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.449943 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.448264 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e563268e-31ea-4ddb-82ff-da5017d5a3fb-etc-kubernetes\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.449943 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.449291 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-ovn-node-metrics-cert\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.449943 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.449685 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/659ae1d5-f1bd-4bec-b2c9-4b5fe3ae0a8d-agent-certs\") pod \"konnectivity-agent-bb6wd\" (UID: \"659ae1d5-f1bd-4bec-b2c9-4b5fe3ae0a8d\") " pod="kube-system/konnectivity-agent-bb6wd" Jul 14 19:10:49.457378 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.457350 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh2xz\" (UniqueName: \"kubernetes.io/projected/e563268e-31ea-4ddb-82ff-da5017d5a3fb-kube-api-access-zh2xz\") pod \"multus-pxrgt\" (UID: \"e563268e-31ea-4ddb-82ff-da5017d5a3fb\") " pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.457378 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.457367 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k676k\" (UniqueName: \"kubernetes.io/projected/dd9dbd61-c491-4f7a-81b7-1769a9e2cb02-kube-api-access-k676k\") pod \"ovnkube-node-tg2x9\" (UID: \"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.457536 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.457350 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p6kq\" (UniqueName: \"kubernetes.io/projected/e0a0b990-bc45-4764-bd78-31069184c58a-kube-api-access-4p6kq\") pod \"node-resolver-q6rws\" (UID: \"e0a0b990-bc45-4764-bd78-31069184c58a\") " pod="openshift-dns/node-resolver-q6rws" Jul 14 19:10:49.457791 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.457774 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghgdc\" (UniqueName: \"kubernetes.io/projected/8d1706ff-a844-42e9-b6d8-628495d88055-kube-api-access-ghgdc\") pod \"network-metrics-daemon-xczln\" (UID: \"8d1706ff-a844-42e9-b6d8-628495d88055\") " pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:10:49.465390 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.465369 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:10:49.512545 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.512496 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-74.ec2.internal" event={"ID":"cc14085ea90483c50f2a94112a007054","Type":"ContainerStarted","Data":"2cf42cb6b8d42a5b4c6ed0878170162235f18c3bdec62266da0429cdeb0ac358"} Jul 14 19:10:49.549601 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549562 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d475a4f-8acf-4cc1-b605-be9c1be74d34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.549820 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec17ab27-ef49-4fae-9006-b236a56a1727-host-slash\") pod \"iptables-alerter-msbnp\" (UID: \"ec17ab27-ef49-4fae-9006-b236a56a1727\") " pod="openshift-network-operator/iptables-alerter-msbnp" Jul 14 19:10:49.549820 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fcgx\" (UniqueName: \"kubernetes.io/projected/70375e8e-ae6d-4ce1-8240-012361054757-kube-api-access-8fcgx\") pod \"node-ca-z4lnk\" (UID: \"70375e8e-ae6d-4ce1-8240-012361054757\") " pod="openshift-image-registry/node-ca-z4lnk" Jul 14 19:10:49.549820 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/81bf32b9-26cc-4396-8d68-9fc1c306e620-sys-fs\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.549820 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549682 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec17ab27-ef49-4fae-9006-b236a56a1727-host-slash\") pod \"iptables-alerter-msbnp\" (UID: \"ec17ab27-ef49-4fae-9006-b236a56a1727\") " pod="openshift-network-operator/iptables-alerter-msbnp" Jul 14 19:10:49.549820 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-sysconfig\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.549820 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d475a4f-8acf-4cc1-b605-be9c1be74d34-cni-binary-copy\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.549820 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549798 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/81bf32b9-26cc-4396-8d68-9fc1c306e620-socket-dir\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.549820 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549815 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/81bf32b9-26cc-4396-8d68-9fc1c306e620-registration-dir\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.549820 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549813 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/81bf32b9-26cc-4396-8d68-9fc1c306e620-sys-fs\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.550962 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2hfg\" (UniqueName: \"kubernetes.io/projected/2d475a4f-8acf-4cc1-b605-be9c1be74d34-kube-api-access-h2hfg\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.550962 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549806 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-sysconfig\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.550962 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549866 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/81bf32b9-26cc-4396-8d68-9fc1c306e620-device-dir\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.550962 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlf7j\" (UniqueName: \"kubernetes.io/projected/5dc12361-b85d-43a1-a491-e4072a9336b1-kube-api-access-tlf7j\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.550962 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549925 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/81bf32b9-26cc-4396-8d68-9fc1c306e620-registration-dir\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.550962 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70375e8e-ae6d-4ce1-8240-012361054757-serviceca\") pod \"node-ca-z4lnk\" (UID: \"70375e8e-ae6d-4ce1-8240-012361054757\") " pod="openshift-image-registry/node-ca-z4lnk" Jul 14 19:10:49.550962 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549952 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/81bf32b9-26cc-4396-8d68-9fc1c306e620-socket-dir\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.550962 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e6c415f-305f-477e-b939-1c2b84a3d770-dbus\") pod \"global-pull-secret-syncer-bv2wz\" (UID: \"5e6c415f-305f-477e-b939-1c2b84a3d770\") " pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:10:49.550962 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.549996 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/81bf32b9-26cc-4396-8d68-9fc1c306e620-device-dir\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.550962 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550008 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-lib-modules\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.550962 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-modprobe-d\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.550962 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550060 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-sys\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.550962 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550080 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-host\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.550962 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550099 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dc12361-b85d-43a1-a491-e4072a9336b1-tmp\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.550962 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550126 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-systemd\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.550962 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550140 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e6c415f-305f-477e-b939-1c2b84a3d770-dbus\") pod \"global-pull-secret-syncer-bv2wz\" (UID: \"5e6c415f-305f-477e-b939-1c2b84a3d770\") " pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:10:49.550962 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550150 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70375e8e-ae6d-4ce1-8240-012361054757-host\") pod \"node-ca-z4lnk\" (UID: \"70375e8e-ae6d-4ce1-8240-012361054757\") " pod="openshift-image-registry/node-ca-z4lnk" Jul 14 19:10:49.551922 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550176 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-lib-modules\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.551922 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550196 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-host\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.551922 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550187 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-modprobe-d\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.551922 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gx7v\" (UniqueName: \"kubernetes.io/projected/81bf32b9-26cc-4396-8d68-9fc1c306e620-kube-api-access-7gx7v\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.551922 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550248 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-kubernetes\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.551922 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550254 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70375e8e-ae6d-4ce1-8240-012361054757-host\") pod \"node-ca-z4lnk\" (UID: \"70375e8e-ae6d-4ce1-8240-012361054757\") " pod="openshift-image-registry/node-ca-z4lnk" Jul 14 19:10:49.551922 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550265 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-tuned\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.551922 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d475a4f-8acf-4cc1-b605-be9c1be74d34-cnibin\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.551922 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550318 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-kubernetes\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.551922 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550218 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-sys\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.551922 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-sysctl-d\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.551922 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550340 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d475a4f-8acf-4cc1-b605-be9c1be74d34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.551922 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550360 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d475a4f-8acf-4cc1-b605-be9c1be74d34-cnibin\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.551922 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550247 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-systemd\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.551922 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x7rx\" (UniqueName: \"kubernetes.io/projected/ec17ab27-ef49-4fae-9006-b236a56a1727-kube-api-access-8x7rx\") pod \"iptables-alerter-msbnp\" (UID: \"ec17ab27-ef49-4fae-9006-b236a56a1727\") " pod="openshift-network-operator/iptables-alerter-msbnp" Jul 14 19:10:49.551922 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550381 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d475a4f-8acf-4cc1-b605-be9c1be74d34-cni-binary-copy\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.551922 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550409 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/81bf32b9-26cc-4396-8d68-9fc1c306e620-etc-selinux\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.552819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550439 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e6c415f-305f-477e-b939-1c2b84a3d770-kubelet-config\") pod \"global-pull-secret-syncer-bv2wz\" (UID: \"5e6c415f-305f-477e-b939-1c2b84a3d770\") " pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:10:49.552819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550467 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-var-lib-kubelet\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.552819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550469 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-sysctl-d\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.552819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550410 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70375e8e-ae6d-4ce1-8240-012361054757-serviceca\") pod \"node-ca-z4lnk\" (UID: \"70375e8e-ae6d-4ce1-8240-012361054757\") " pod="openshift-image-registry/node-ca-z4lnk" Jul 14 19:10:49.552819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550494 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d475a4f-8acf-4cc1-b605-be9c1be74d34-os-release\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.552819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550524 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/81bf32b9-26cc-4396-8d68-9fc1c306e620-etc-selinux\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.552819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550548 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-var-lib-kubelet\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.552819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550549 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d475a4f-8acf-4cc1-b605-be9c1be74d34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.552819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550558 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d475a4f-8acf-4cc1-b605-be9c1be74d34-os-release\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.552819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret\") pod \"global-pull-secret-syncer-bv2wz\" (UID: \"5e6c415f-305f-477e-b939-1c2b84a3d770\") " pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:10:49.552819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550614 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d475a4f-8acf-4cc1-b605-be9c1be74d34-system-cni-dir\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.552819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec17ab27-ef49-4fae-9006-b236a56a1727-iptables-alerter-script\") pod \"iptables-alerter-msbnp\" (UID: \"ec17ab27-ef49-4fae-9006-b236a56a1727\") " pod="openshift-network-operator/iptables-alerter-msbnp" Jul 14 19:10:49.552819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550648 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e6c415f-305f-477e-b939-1c2b84a3d770-kubelet-config\") pod \"global-pull-secret-syncer-bv2wz\" (UID: \"5e6c415f-305f-477e-b939-1c2b84a3d770\") " pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:10:49.552819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/81bf32b9-26cc-4396-8d68-9fc1c306e620-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.552819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550696 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d475a4f-8acf-4cc1-b605-be9c1be74d34-system-cni-dir\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.552819 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:49.550707 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:10:49.552819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550715 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d475a4f-8acf-4cc1-b605-be9c1be74d34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.553469 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-sysctl-conf\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.553469 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550748 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-run\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.553469 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d475a4f-8acf-4cc1-b605-be9c1be74d34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.553469 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550776 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/81bf32b9-26cc-4396-8d68-9fc1c306e620-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.553469 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:49.550833 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret podName:5e6c415f-305f-477e-b939-1c2b84a3d770 nodeName:}" failed. No retries permitted until 2026-07-14 19:10:50.050812173 +0000 UTC m=+3.164062029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret") pod "global-pull-secret-syncer-bv2wz" (UID: "5e6c415f-305f-477e-b939-1c2b84a3d770") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:10:49.553469 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550838 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-run\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.553469 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550896 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwg6s\" (UniqueName: \"kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s\") pod \"network-check-target-tgldx\" (UID: \"67bb23c7-9bee-4350-a4bf-0da1589fbce0\") " pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:10:49.553469 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.550960 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-sysctl-conf\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.553469 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.551172 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec17ab27-ef49-4fae-9006-b236a56a1727-iptables-alerter-script\") pod \"iptables-alerter-msbnp\" (UID: \"ec17ab27-ef49-4fae-9006-b236a56a1727\") " pod="openshift-network-operator/iptables-alerter-msbnp" Jul 14 19:10:49.553469 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.551209 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d475a4f-8acf-4cc1-b605-be9c1be74d34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.553469 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.552508 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5dc12361-b85d-43a1-a491-e4072a9336b1-etc-tuned\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.553469 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.553168 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dc12361-b85d-43a1-a491-e4072a9336b1-tmp\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.564644 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:49.564574 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:10:49.564644 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:49.564599 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:10:49.564644 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:49.564614 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cwg6s for pod openshift-network-diagnostics/network-check-target-tgldx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:10:49.564908 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:49.564715 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s podName:67bb23c7-9bee-4350-a4bf-0da1589fbce0 nodeName:}" failed. No retries permitted until 2026-07-14 19:10:50.064694574 +0000 UTC m=+3.177944443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cwg6s" (UniqueName: "kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s") pod "network-check-target-tgldx" (UID: "67bb23c7-9bee-4350-a4bf-0da1589fbce0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:10:49.565557 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.565529 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fcgx\" (UniqueName: \"kubernetes.io/projected/70375e8e-ae6d-4ce1-8240-012361054757-kube-api-access-8fcgx\") pod \"node-ca-z4lnk\" (UID: \"70375e8e-ae6d-4ce1-8240-012361054757\") " pod="openshift-image-registry/node-ca-z4lnk" Jul 14 19:10:49.567042 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.567012 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlf7j\" (UniqueName: \"kubernetes.io/projected/5dc12361-b85d-43a1-a491-e4072a9336b1-kube-api-access-tlf7j\") pod \"tuned-8k4j8\" (UID: \"5dc12361-b85d-43a1-a491-e4072a9336b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.567532 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.567511 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x7rx\" (UniqueName: \"kubernetes.io/projected/ec17ab27-ef49-4fae-9006-b236a56a1727-kube-api-access-8x7rx\") pod \"iptables-alerter-msbnp\" (UID: \"ec17ab27-ef49-4fae-9006-b236a56a1727\") " pod="openshift-network-operator/iptables-alerter-msbnp" Jul 14 19:10:49.567790 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.567771 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2hfg\" (UniqueName: \"kubernetes.io/projected/2d475a4f-8acf-4cc1-b605-be9c1be74d34-kube-api-access-h2hfg\") pod \"multus-additional-cni-plugins-f7zqk\" (UID: \"2d475a4f-8acf-4cc1-b605-be9c1be74d34\") " pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.568490 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.568462 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gx7v\" (UniqueName: \"kubernetes.io/projected/81bf32b9-26cc-4396-8d68-9fc1c306e620-kube-api-access-7gx7v\") pod \"aws-ebs-csi-driver-node-s7nct\" (UID: \"81bf32b9-26cc-4396-8d68-9fc1c306e620\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.637024 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.636988 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:10:49.645856 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.645834 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q6rws" Jul 14 19:10:49.647736 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:49.647706 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9dbd61_c491_4f7a_81b7_1769a9e2cb02.slice/crio-c1a55d26ab16b4391befab79d6630b72aeddf8b44e3004c9f5bfb58b135786ac WatchSource:0}: Error finding container c1a55d26ab16b4391befab79d6630b72aeddf8b44e3004c9f5bfb58b135786ac: Status 404 returned error can't find the container with id c1a55d26ab16b4391befab79d6630b72aeddf8b44e3004c9f5bfb58b135786ac Jul 14 19:10:49.652397 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.652372 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bb6wd" Jul 14 19:10:49.654556 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:49.654534 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a0b990_bc45_4764_bd78_31069184c58a.slice/crio-18a132ac198e638092ec76c3779581770cbb712ffad18a747203970333b88248 WatchSource:0}: Error finding container 18a132ac198e638092ec76c3779581770cbb712ffad18a747203970333b88248: Status 404 returned error can't find the container with id 18a132ac198e638092ec76c3779581770cbb712ffad18a747203970333b88248 Jul 14 19:10:49.659855 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.659834 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pxrgt" Jul 14 19:10:49.665757 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.665735 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" Jul 14 19:10:49.671462 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.671442 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f7zqk" Jul 14 19:10:49.672111 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:49.672087 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659ae1d5_f1bd_4bec_b2c9_4b5fe3ae0a8d.slice/crio-02b35ebbe64f89161a3cff3bac6aba1603c8004c77e8d0239cab03c9c481ac25 WatchSource:0}: Error finding container 02b35ebbe64f89161a3cff3bac6aba1603c8004c77e8d0239cab03c9c481ac25: Status 404 returned error can't find the container with id 02b35ebbe64f89161a3cff3bac6aba1603c8004c77e8d0239cab03c9c481ac25 Jul 14 19:10:49.672595 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:49.672571 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode563268e_31ea_4ddb_82ff_da5017d5a3fb.slice/crio-4665770e4ff682f86840a395c6baa23c8d8c22988e392d64fc3cf9261de02748 WatchSource:0}: Error finding container 4665770e4ff682f86840a395c6baa23c8d8c22988e392d64fc3cf9261de02748: Status 404 returned error can't find the container with id 4665770e4ff682f86840a395c6baa23c8d8c22988e392d64fc3cf9261de02748 Jul 14 19:10:49.675629 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:49.675573 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471487e6885ab16f5c27e006ead6368f.slice/crio-076eee5abc8602ebb572b8c3cc32a4e439e5dec5eee2bdabb0c8f1026b7b9af9 WatchSource:0}: Error finding container 076eee5abc8602ebb572b8c3cc32a4e439e5dec5eee2bdabb0c8f1026b7b9af9: Status 404 returned error can't find the container with id 076eee5abc8602ebb572b8c3cc32a4e439e5dec5eee2bdabb0c8f1026b7b9af9 Jul 14 19:10:49.678143 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.678125 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-msbnp" Jul 14 19:10:49.679121 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:49.679091 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc12361_b85d_43a1_a491_e4072a9336b1.slice/crio-0906fc6788cab6e2c3f1d9dc3ad7fbbbbb5878f7f81f73681829da59aaa42733 WatchSource:0}: Error finding container 0906fc6788cab6e2c3f1d9dc3ad7fbbbbb5878f7f81f73681829da59aaa42733: Status 404 returned error can't find the container with id 0906fc6788cab6e2c3f1d9dc3ad7fbbbbb5878f7f81f73681829da59aaa42733 Jul 14 19:10:49.683220 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:49.683175 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d475a4f_8acf_4cc1_b605_be9c1be74d34.slice/crio-27581606ca57ad38e14b21ff966fb0ac92cbd3d01e09a743534573d2ea2fff6d WatchSource:0}: Error finding container 27581606ca57ad38e14b21ff966fb0ac92cbd3d01e09a743534573d2ea2fff6d: Status 404 returned error can't find the container with id 27581606ca57ad38e14b21ff966fb0ac92cbd3d01e09a743534573d2ea2fff6d Jul 14 19:10:49.684404 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.684385 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z4lnk" Jul 14 19:10:49.689201 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.689180 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" Jul 14 19:10:49.689722 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:49.689646 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec17ab27_ef49_4fae_9006_b236a56a1727.slice/crio-2881399b93e5ef00a41aa95cbb7fbd17b7438fd2a980825b4a6231553a000b5e WatchSource:0}: Error finding container 2881399b93e5ef00a41aa95cbb7fbd17b7438fd2a980825b4a6231553a000b5e: Status 404 returned error can't find the container with id 2881399b93e5ef00a41aa95cbb7fbd17b7438fd2a980825b4a6231553a000b5e Jul 14 19:10:49.694001 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:49.693979 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70375e8e_ae6d_4ce1_8240_012361054757.slice/crio-3e05ff6ee79338137a6016ad62c146f490b41e9ac208434efe4361bf7691efc3 WatchSource:0}: Error finding container 3e05ff6ee79338137a6016ad62c146f490b41e9ac208434efe4361bf7691efc3: Status 404 returned error can't find the container with id 3e05ff6ee79338137a6016ad62c146f490b41e9ac208434efe4361bf7691efc3 Jul 14 19:10:49.698345 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:10:49.698319 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81bf32b9_26cc_4396_8d68_9fc1c306e620.slice/crio-b7e018d1087b6106f190f80acc021723dc04d28d1ff9349efc48a54ebffc3cbc WatchSource:0}: Error finding container b7e018d1087b6106f190f80acc021723dc04d28d1ff9349efc48a54ebffc3cbc: Status 404 returned error can't find the container with id b7e018d1087b6106f190f80acc021723dc04d28d1ff9349efc48a54ebffc3cbc Jul 14 19:10:49.953880 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:49.953788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs\") pod \"network-metrics-daemon-xczln\" (UID: \"8d1706ff-a844-42e9-b6d8-628495d88055\") " pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:10:49.954073 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:49.953908 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:10:49.954073 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:49.953961 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs podName:8d1706ff-a844-42e9-b6d8-628495d88055 nodeName:}" failed. No retries permitted until 2026-07-14 19:10:50.953947421 +0000 UTC m=+4.067197270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs") pod "network-metrics-daemon-xczln" (UID: "8d1706ff-a844-42e9-b6d8-628495d88055") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:10:50.055171 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:50.055130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret\") pod \"global-pull-secret-syncer-bv2wz\" (UID: \"5e6c415f-305f-477e-b939-1c2b84a3d770\") " pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:10:50.055356 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:50.055291 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:10:50.055356 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:50.055353 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret podName:5e6c415f-305f-477e-b939-1c2b84a3d770 nodeName:}" failed. No retries permitted until 2026-07-14 19:10:51.055338835 +0000 UTC m=+4.168588685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret") pod "global-pull-secret-syncer-bv2wz" (UID: "5e6c415f-305f-477e-b939-1c2b84a3d770") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:10:50.156034 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:50.155982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwg6s\" (UniqueName: \"kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s\") pod \"network-check-target-tgldx\" (UID: \"67bb23c7-9bee-4350-a4bf-0da1589fbce0\") " pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:10:50.156260 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:50.156169 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:10:50.156260 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:50.156196 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:10:50.156260 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:50.156210 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cwg6s for pod openshift-network-diagnostics/network-check-target-tgldx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:10:50.156447 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:50.156280 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s podName:67bb23c7-9bee-4350-a4bf-0da1589fbce0 nodeName:}" failed. No retries permitted until 2026-07-14 19:10:51.15625977 +0000 UTC m=+4.269509625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cwg6s" (UniqueName: "kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s") pod "network-check-target-tgldx" (UID: "67bb23c7-9bee-4350-a4bf-0da1589fbce0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:10:50.391508 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:50.391406 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 19:05:49 +0000 UTC" deadline="2028-04-15 04:10:27.138924648 +0000 UTC" Jul 14 19:10:50.391508 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:50.391449 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15368h59m36.747480556s" Jul 14 19:10:50.517157 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:50.517115 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pxrgt" event={"ID":"e563268e-31ea-4ddb-82ff-da5017d5a3fb","Type":"ContainerStarted","Data":"4665770e4ff682f86840a395c6baa23c8d8c22988e392d64fc3cf9261de02748"} Jul 14 19:10:50.518907 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:50.518838 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bb6wd" event={"ID":"659ae1d5-f1bd-4bec-b2c9-4b5fe3ae0a8d","Type":"ContainerStarted","Data":"02b35ebbe64f89161a3cff3bac6aba1603c8004c77e8d0239cab03c9c481ac25"} Jul 14 19:10:50.520432 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:50.520367 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q6rws" event={"ID":"e0a0b990-bc45-4764-bd78-31069184c58a","Type":"ContainerStarted","Data":"18a132ac198e638092ec76c3779581770cbb712ffad18a747203970333b88248"} Jul 14 19:10:50.523607 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:50.523538 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" event={"ID":"81bf32b9-26cc-4396-8d68-9fc1c306e620","Type":"ContainerStarted","Data":"b7e018d1087b6106f190f80acc021723dc04d28d1ff9349efc48a54ebffc3cbc"} Jul 14 19:10:50.526022 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:50.525995 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-msbnp" event={"ID":"ec17ab27-ef49-4fae-9006-b236a56a1727","Type":"ContainerStarted","Data":"2881399b93e5ef00a41aa95cbb7fbd17b7438fd2a980825b4a6231553a000b5e"} Jul 14 19:10:50.528745 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:50.528719 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" event={"ID":"5dc12361-b85d-43a1-a491-e4072a9336b1","Type":"ContainerStarted","Data":"0906fc6788cab6e2c3f1d9dc3ad7fbbbbb5878f7f81f73681829da59aaa42733"} Jul 14 19:10:50.530653 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:50.530629 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" event={"ID":"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02","Type":"ContainerStarted","Data":"c1a55d26ab16b4391befab79d6630b72aeddf8b44e3004c9f5bfb58b135786ac"} Jul 14 19:10:50.532085 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:50.532027 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z4lnk" event={"ID":"70375e8e-ae6d-4ce1-8240-012361054757","Type":"ContainerStarted","Data":"3e05ff6ee79338137a6016ad62c146f490b41e9ac208434efe4361bf7691efc3"} Jul 14 19:10:50.536152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:50.534360 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f7zqk" event={"ID":"2d475a4f-8acf-4cc1-b605-be9c1be74d34","Type":"ContainerStarted","Data":"27581606ca57ad38e14b21ff966fb0ac92cbd3d01e09a743534573d2ea2fff6d"} Jul 14 19:10:50.536297 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:50.536154 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal" event={"ID":"471487e6885ab16f5c27e006ead6368f","Type":"ContainerStarted","Data":"076eee5abc8602ebb572b8c3cc32a4e439e5dec5eee2bdabb0c8f1026b7b9af9"} Jul 14 19:10:50.962842 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:50.962804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs\") pod \"network-metrics-daemon-xczln\" (UID: \"8d1706ff-a844-42e9-b6d8-628495d88055\") " pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:10:50.963035 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:50.962952 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:10:50.963108 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:50.963032 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs podName:8d1706ff-a844-42e9-b6d8-628495d88055 nodeName:}" failed. No retries permitted until 2026-07-14 19:10:52.963013885 +0000 UTC m=+6.076263739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs") pod "network-metrics-daemon-xczln" (UID: "8d1706ff-a844-42e9-b6d8-628495d88055") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:10:51.063562 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:51.063219 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret\") pod \"global-pull-secret-syncer-bv2wz\" (UID: \"5e6c415f-305f-477e-b939-1c2b84a3d770\") " pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:10:51.063562 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:51.063430 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:10:51.063562 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:51.063493 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret podName:5e6c415f-305f-477e-b939-1c2b84a3d770 nodeName:}" failed. No retries permitted until 2026-07-14 19:10:53.06347496 +0000 UTC m=+6.176724812 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret") pod "global-pull-secret-syncer-bv2wz" (UID: "5e6c415f-305f-477e-b939-1c2b84a3d770") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:10:51.164270 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:51.164228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwg6s\" (UniqueName: \"kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s\") pod \"network-check-target-tgldx\" (UID: \"67bb23c7-9bee-4350-a4bf-0da1589fbce0\") " pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:10:51.164492 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:51.164430 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:10:51.164492 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:51.164454 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:10:51.164492 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:51.164468 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cwg6s for pod openshift-network-diagnostics/network-check-target-tgldx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:10:51.164681 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:51.164532 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s podName:67bb23c7-9bee-4350-a4bf-0da1589fbce0 nodeName:}" failed. No retries permitted until 2026-07-14 19:10:53.164513171 +0000 UTC m=+6.277763027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cwg6s" (UniqueName: "kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s") pod "network-check-target-tgldx" (UID: "67bb23c7-9bee-4350-a4bf-0da1589fbce0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:10:51.392624 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:51.392487 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 19:05:49 +0000 UTC" deadline="2027-12-13 09:24:09.362405929 +0000 UTC" Jul 14 19:10:51.392624 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:51.392550 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12398h13m17.969859631s" Jul 14 19:10:51.510339 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:51.510240 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:10:51.510339 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:51.510243 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:10:51.510549 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:51.510361 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:10:51.510549 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:51.510387 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:10:51.510549 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:51.510465 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:10:51.510742 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:51.510543 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:10:52.982720 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:52.980431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs\") pod \"network-metrics-daemon-xczln\" (UID: \"8d1706ff-a844-42e9-b6d8-628495d88055\") " pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:10:52.982720 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:52.980891 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:10:52.982720 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:52.980964 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs podName:8d1706ff-a844-42e9-b6d8-628495d88055 nodeName:}" failed. No retries permitted until 2026-07-14 19:10:56.980941894 +0000 UTC m=+10.094191743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs") pod "network-metrics-daemon-xczln" (UID: "8d1706ff-a844-42e9-b6d8-628495d88055") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:10:53.081482 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:53.081097 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret\") pod \"global-pull-secret-syncer-bv2wz\" (UID: \"5e6c415f-305f-477e-b939-1c2b84a3d770\") " pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:10:53.081482 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:53.081262 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:10:53.081482 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:53.081333 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret podName:5e6c415f-305f-477e-b939-1c2b84a3d770 nodeName:}" failed. No retries permitted until 2026-07-14 19:10:57.081305273 +0000 UTC m=+10.194555126 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret") pod "global-pull-secret-syncer-bv2wz" (UID: "5e6c415f-305f-477e-b939-1c2b84a3d770") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:10:53.182461 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:53.182427 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwg6s\" (UniqueName: \"kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s\") pod \"network-check-target-tgldx\" (UID: \"67bb23c7-9bee-4350-a4bf-0da1589fbce0\") " pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:10:53.182723 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:53.182671 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:10:53.182723 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:53.182694 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:10:53.182723 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:53.182707 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cwg6s for pod openshift-network-diagnostics/network-check-target-tgldx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:10:53.182911 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:53.182762 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s podName:67bb23c7-9bee-4350-a4bf-0da1589fbce0 nodeName:}" failed. No retries permitted until 2026-07-14 19:10:57.182744226 +0000 UTC m=+10.295994097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cwg6s" (UniqueName: "kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s") pod "network-check-target-tgldx" (UID: "67bb23c7-9bee-4350-a4bf-0da1589fbce0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:10:53.511031 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:53.510990 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:10:53.511221 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:53.511143 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:10:53.511518 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:53.511501 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:10:53.511637 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:53.511624 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:10:53.511701 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:53.511668 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:10:53.511753 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:53.511736 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:10:53.550942 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:53.550904 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bb6wd" event={"ID":"659ae1d5-f1bd-4bec-b2c9-4b5fe3ae0a8d","Type":"ContainerStarted","Data":"05c0fbd7327a4f28be6ab654473143c8bbf7675c0d135a9fd7f942137aa7db36"} Jul 14 19:10:53.904426 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:53.904283 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bb6wd" Jul 14 19:10:53.905048 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:53.904832 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bb6wd" Jul 14 19:10:53.923078 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:53.923009 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bb6wd" podStartSLOduration=3.91402769 podStartE2EDuration="6.92298835s" totalImagesPullingTime="3.00896066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:10:53.568702208 +0000 UTC m=+6.681952079" watchObservedRunningTime="2026-07-14 19:10:53.92298835 +0000 UTC m=+7.036238220" Jul 14 19:10:55.514785 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:55.513497 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:10:55.514785 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:55.513734 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:10:55.514785 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:55.514309 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:10:55.514785 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:55.514466 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:10:55.514785 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:55.514610 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:10:55.514785 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:55.514737 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:10:55.558510 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:55.558443 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 14 19:10:56.562864 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:56.562829 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal" event={"ID":"471487e6885ab16f5c27e006ead6368f","Type":"ContainerStarted","Data":"eac6eaa6a3d747d6e7f7689c3fd449f04a998306e47c7eae3961376bc98e035d"} Jul 14 19:10:57.015699 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:57.015561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs\") pod \"network-metrics-daemon-xczln\" (UID: \"8d1706ff-a844-42e9-b6d8-628495d88055\") " pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:10:57.015699 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:57.015695 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:10:57.015902 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:57.015764 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs podName:8d1706ff-a844-42e9-b6d8-628495d88055 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:05.015745991 +0000 UTC m=+18.128995843 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs") pod "network-metrics-daemon-xczln" (UID: "8d1706ff-a844-42e9-b6d8-628495d88055") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:10:57.116991 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:57.116953 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret\") pod \"global-pull-secret-syncer-bv2wz\" (UID: \"5e6c415f-305f-477e-b939-1c2b84a3d770\") " pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:10:57.117232 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:57.117112 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:10:57.117232 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:57.117187 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret podName:5e6c415f-305f-477e-b939-1c2b84a3d770 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:05.117167523 +0000 UTC m=+18.230417390 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret") pod "global-pull-secret-syncer-bv2wz" (UID: "5e6c415f-305f-477e-b939-1c2b84a3d770") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:10:57.217576 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:57.217536 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwg6s\" (UniqueName: \"kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s\") pod \"network-check-target-tgldx\" (UID: \"67bb23c7-9bee-4350-a4bf-0da1589fbce0\") " pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:10:57.217778 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:57.217756 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:10:57.217844 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:57.217789 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:10:57.217844 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:57.217804 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cwg6s for pod openshift-network-diagnostics/network-check-target-tgldx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:10:57.217956 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:57.217872 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s podName:67bb23c7-9bee-4350-a4bf-0da1589fbce0 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:05.21785262 +0000 UTC m=+18.331102474 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cwg6s" (UniqueName: "kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s") pod "network-check-target-tgldx" (UID: "67bb23c7-9bee-4350-a4bf-0da1589fbce0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:10:57.510574 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:57.510499 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:10:57.510748 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:57.510633 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:10:57.510748 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:57.510652 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:10:57.510748 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:57.510739 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:10:57.510911 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:57.510801 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:10:57.510911 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:57.510876 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:10:58.569332 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:58.569232 2576 generic.go:356] "Generic (PLEG): container finished" podID="471487e6885ab16f5c27e006ead6368f" containerID="eac6eaa6a3d747d6e7f7689c3fd449f04a998306e47c7eae3961376bc98e035d" exitCode=0 Jul 14 19:10:58.569332 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:58.569282 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal" event={"ID":"471487e6885ab16f5c27e006ead6368f","Type":"ContainerDied","Data":"eac6eaa6a3d747d6e7f7689c3fd449f04a998306e47c7eae3961376bc98e035d"} Jul 14 19:10:59.509126 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:59.508728 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:10:59.509126 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:59.508751 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:10:59.509126 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:59.508888 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:10:59.509126 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:10:59.508949 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:10:59.509126 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:59.509019 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:10:59.509963 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:10:59.509099 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:11:01.509290 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:01.509250 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:01.509762 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:01.509250 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:11:01.509762 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:01.509372 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:11:01.509762 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:01.509250 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:01.509762 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:01.509444 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:11:01.509762 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:01.509512 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:11:03.509536 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:03.509495 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:11:03.509536 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:03.509528 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:03.509536 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:03.509495 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:03.510109 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:03.509654 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:11:03.510109 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:03.509824 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:11:03.510109 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:03.509919 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:11:05.078076 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:05.077999 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs\") pod \"network-metrics-daemon-xczln\" (UID: \"8d1706ff-a844-42e9-b6d8-628495d88055\") " pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:11:05.078526 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:05.078122 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:11:05.078526 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:05.078187 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs podName:8d1706ff-a844-42e9-b6d8-628495d88055 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:21.078168 +0000 UTC m=+34.191417853 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs") pod "network-metrics-daemon-xczln" (UID: "8d1706ff-a844-42e9-b6d8-628495d88055") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:11:05.178484 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:05.178451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret\") pod \"global-pull-secret-syncer-bv2wz\" (UID: \"5e6c415f-305f-477e-b939-1c2b84a3d770\") " pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:05.178667 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:05.178615 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:11:05.178732 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:05.178685 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret podName:5e6c415f-305f-477e-b939-1c2b84a3d770 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:21.178664534 +0000 UTC m=+34.291914389 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret") pod "global-pull-secret-syncer-bv2wz" (UID: "5e6c415f-305f-477e-b939-1c2b84a3d770") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:11:05.279272 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:05.279242 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwg6s\" (UniqueName: \"kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s\") pod \"network-check-target-tgldx\" (UID: \"67bb23c7-9bee-4350-a4bf-0da1589fbce0\") " pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:05.279443 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:05.279395 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:11:05.279443 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:05.279415 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:11:05.279443 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:05.279426 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cwg6s for pod openshift-network-diagnostics/network-check-target-tgldx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:11:05.279575 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:05.279488 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s podName:67bb23c7-9bee-4350-a4bf-0da1589fbce0 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:21.279469688 +0000 UTC m=+34.392719537 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cwg6s" (UniqueName: "kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s") pod "network-check-target-tgldx" (UID: "67bb23c7-9bee-4350-a4bf-0da1589fbce0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:11:05.509774 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:05.509689 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:11:05.509774 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:05.509726 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:05.509774 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:05.509691 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:05.510040 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:05.509842 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:11:05.510232 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:05.510203 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:11:05.510361 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:05.510310 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:11:07.510036 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:07.510003 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:07.510464 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:07.510097 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:11:07.510464 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:07.510120 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:11:07.510464 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:07.510143 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:07.510464 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:07.510243 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:11:07.510464 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:07.510325 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:11:09.392446 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:09.392416 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bb6wd" Jul 14 19:11:09.393264 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:09.393241 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bb6wd" Jul 14 19:11:09.509787 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:09.509397 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:11:09.509787 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:09.509744 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:11:09.509787 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:09.509784 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:09.510007 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:09.509861 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:11:09.510007 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:09.509921 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:09.510007 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:09.509991 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:11:09.589739 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:09.589705 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" event={"ID":"5dc12361-b85d-43a1-a491-e4072a9336b1","Type":"ContainerStarted","Data":"a04b859339a7dcffe974bc9948a60147b0ed61477a269c797268536373c51551"} Jul 14 19:11:09.591738 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:09.591438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z4lnk" event={"ID":"70375e8e-ae6d-4ce1-8240-012361054757","Type":"ContainerStarted","Data":"4529479b63fd3401dbc006b8fff159c445547dc9b508e2975f385158a1b8af01"} Jul 14 19:11:09.593656 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:09.593245 2576 generic.go:356] "Generic (PLEG): container finished" podID="2d475a4f-8acf-4cc1-b605-be9c1be74d34" containerID="0e5c87ebb608b900d0fc6d9bedb3e85dd7a8ec4bf45026f0a0575a2c67469fe7" exitCode=0 Jul 14 19:11:09.593656 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:09.593358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f7zqk" event={"ID":"2d475a4f-8acf-4cc1-b605-be9c1be74d34","Type":"ContainerDied","Data":"0e5c87ebb608b900d0fc6d9bedb3e85dd7a8ec4bf45026f0a0575a2c67469fe7"} Jul 14 19:11:09.596682 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:09.596658 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal" event={"ID":"471487e6885ab16f5c27e006ead6368f","Type":"ContainerStarted","Data":"7029438fbdcf8566e40681a2621aea7b65d3a4704848600bfdd48c19dbd5c809"} Jul 14 19:11:09.598180 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:09.598160 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pxrgt" event={"ID":"e563268e-31ea-4ddb-82ff-da5017d5a3fb","Type":"ContainerStarted","Data":"3219e00369667f2d5f03ca961cf440d7abac5cdff2252581e844f76abb755c8b"} Jul 14 19:11:09.599606 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:09.599569 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-74.ec2.internal" event={"ID":"cc14085ea90483c50f2a94112a007054","Type":"ContainerStarted","Data":"713a1500a30d9d657d3a5e38740f23a3525438b73a678b16ee1757d9c4f7fc81"} Jul 14 19:11:09.601139 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:09.601114 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" event={"ID":"81bf32b9-26cc-4396-8d68-9fc1c306e620","Type":"ContainerStarted","Data":"ae8cbd3688221520feb51377a213dd61492329a15fb8c12c7c6d6a15d3e040df"} Jul 14 19:11:09.618274 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:09.618224 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8k4j8" podStartSLOduration=2.02586608 podStartE2EDuration="21.618206953s" totalImagesPullingTime="19.592340873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:09.617438163 +0000 UTC m=+22.730688032" watchObservedRunningTime="2026-07-14 19:11:09.618206953 +0000 UTC m=+22.731456823" Jul 14 19:11:09.634155 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:09.634097 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z4lnk" podStartSLOduration=6.478215295 podStartE2EDuration="21.634079353s" totalImagesPullingTime="15.155864058s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:09.633733319 +0000 UTC m=+22.746983188" watchObservedRunningTime="2026-07-14 19:11:09.634079353 +0000 UTC m=+22.747329224" Jul 14 19:11:09.649722 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:09.649667 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-74.ec2.internal" podStartSLOduration=20.649653004 podStartE2EDuration="20.649653004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:09.648983639 +0000 UTC m=+22.762233510" watchObservedRunningTime="2026-07-14 19:11:09.649653004 +0000 UTC m=+22.762902874" Jul 14 19:11:09.690094 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:09.690041 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pxrgt" podStartSLOduration=2.857638856 podStartE2EDuration="22.690024669s" totalImagesPullingTime="19.832385813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:09.689958016 +0000 UTC m=+22.803207898" watchObservedRunningTime="2026-07-14 19:11:09.690024669 +0000 UTC m=+22.803274539" Jul 14 19:11:09.690452 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:09.690432 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-74.ec2.internal" podStartSLOduration=20.690426376 podStartE2EDuration="20.690426376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:09.667746737 +0000 UTC m=+22.780996608" watchObservedRunningTime="2026-07-14 19:11:09.690426376 +0000 UTC m=+22.803676246" Jul 14 19:11:10.604352 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:10.604323 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-msbnp" event={"ID":"ec17ab27-ef49-4fae-9006-b236a56a1727","Type":"ContainerStarted","Data":"b2b8a04a876abeb42a071cff04e4d885f07a1001ee24781e506468643a8f1a14"} Jul 14 19:11:10.605997 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:10.605978 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:11:10.606324 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:10.606303 2576 generic.go:356] "Generic (PLEG): container finished" podID="dd9dbd61-c491-4f7a-81b7-1769a9e2cb02" containerID="2cc870231f21ce1cfa2ef3c98be3ec04c225ba75548fa803f538db34ab03a877" exitCode=1 Jul 14 19:11:10.606413 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:10.606374 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" event={"ID":"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02","Type":"ContainerDied","Data":"2cc870231f21ce1cfa2ef3c98be3ec04c225ba75548fa803f538db34ab03a877"} Jul 14 19:11:10.606473 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:10.606455 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" event={"ID":"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02","Type":"ContainerStarted","Data":"3f0ccadd0f98e1ea82feecbe115dc18469bcf01a319de357c97e18532ae15d81"} Jul 14 19:11:10.610313 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:10.610146 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q6rws" event={"ID":"e0a0b990-bc45-4764-bd78-31069184c58a","Type":"ContainerStarted","Data":"cd3aaad280cb928b3c38701a7f7df53fa638d3cf15f2fa4a8ce4603a3acc8218"} Jul 14 19:11:10.626948 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:10.626679 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q6rws" podStartSLOduration=4.010801872 podStartE2EDuration="23.626656509s" totalImagesPullingTime="19.615854637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:10.626187198 +0000 UTC m=+23.739437069" watchObservedRunningTime="2026-07-14 19:11:10.626656509 +0000 UTC m=+23.739906379" Jul 14 19:11:11.509476 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:11.509441 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:11.509713 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:11.509454 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:11:11.509713 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:11.509568 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:11:11.509713 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:11.509598 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:11.509868 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:11.509797 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:11:11.509868 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:11.509817 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:11:11.615778 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:11.615753 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:11:11.616409 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:11.616203 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" event={"ID":"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02","Type":"ContainerStarted","Data":"9418a4027f1ea89c54e73da3cf940ec8ccd71c9c11862cb408f442fb03e8fbc2"} Jul 14 19:11:11.616409 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:11.616295 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" event={"ID":"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02","Type":"ContainerStarted","Data":"36925ea46514a1a2fc1659bb8b973f1326b51edb4abcb9aa3c4b8378ad806c5b"} Jul 14 19:11:11.616409 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:11.616341 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" event={"ID":"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02","Type":"ContainerStarted","Data":"7209b0a1b30e533a877ce3218614439c3ca67b0006f93d6a5c9015d37763f960"} Jul 14 19:11:11.616409 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:11.616385 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" event={"ID":"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02","Type":"ContainerStarted","Data":"8e977d9355ccee44d350d7bb40102b1a5a1adcc8ba3d40dbe8e0a71e35b1511f"} Jul 14 19:11:11.631116 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:11.631071 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-msbnp" podStartSLOduration=12.396014725 podStartE2EDuration="23.63105247s" totalImagesPullingTime="11.235037745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:11.630887717 +0000 UTC m=+24.744137588" watchObservedRunningTime="2026-07-14 19:11:11.63105247 +0000 UTC m=+24.744302342" Jul 14 19:11:11.767857 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:11.767778 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 19:11:12.424943 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:12.424789 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T19:11:11.767804244Z","UUID":"ac25447c-df0e-4287-8cd7-54f9d9b6a482","Handler":null,"Name":"","Endpoint":""} Jul 14 19:11:12.426326 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:12.426304 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 19:11:12.426453 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:12.426347 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 19:11:12.620976 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:12.620942 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" event={"ID":"81bf32b9-26cc-4396-8d68-9fc1c306e620","Type":"ContainerStarted","Data":"925bae7c36f995cb4a5d2977da31d13c8f7a9e271b47301f2ac2f6f309c6f974"} Jul 14 19:11:13.509941 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:13.509712 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:13.510112 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:13.509711 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:11:13.510112 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:13.509989 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:11:13.510112 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:13.509721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:13.510259 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:13.510110 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:11:13.510259 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:13.510170 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:11:14.628420 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:14.628392 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:11:14.629066 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:14.628805 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" event={"ID":"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02","Type":"ContainerStarted","Data":"46030a2f0fe8bfd45f7ed07d9e245d9bc09bf0e4a76a4fbc0fcaa63c16b875e8"} Jul 14 19:11:14.630500 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:14.630476 2576 generic.go:356] "Generic (PLEG): container finished" podID="2d475a4f-8acf-4cc1-b605-be9c1be74d34" containerID="b2a18f264f13aab0c09931684ecc2fbabd259334603c2c33bfdfc5ac80b20a42" exitCode=0 Jul 14 19:11:14.630624 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:14.630561 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f7zqk" event={"ID":"2d475a4f-8acf-4cc1-b605-be9c1be74d34","Type":"ContainerDied","Data":"b2a18f264f13aab0c09931684ecc2fbabd259334603c2c33bfdfc5ac80b20a42"} Jul 14 19:11:14.632595 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:14.632564 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" event={"ID":"81bf32b9-26cc-4396-8d68-9fc1c306e620","Type":"ContainerStarted","Data":"39c4b8ff375506e7989db8ad73b56a9e8561c150000722f9d16e7b6d1e86cf4d"} Jul 14 19:11:14.670371 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:14.670329 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7nct" podStartSLOduration=2.2706405370000002 podStartE2EDuration="26.670315576s" totalImagesPullingTime="24.399675039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:14.669994458 +0000 UTC m=+27.783244328" watchObservedRunningTime="2026-07-14 19:11:14.670315576 +0000 UTC m=+27.783565456" Jul 14 19:11:15.508744 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:15.508709 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:11:15.508917 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:15.508715 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:15.508917 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:15.508823 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:11:15.508991 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:15.508910 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:11:15.508991 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:15.508715 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:15.509051 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:15.509012 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:11:16.640226 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:16.640049 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:11:16.640764 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:16.640574 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" event={"ID":"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02","Type":"ContainerStarted","Data":"c78bfbd0f576ff9455d622072c5f7ac212fa9c70442eadaa4001d069182e2b03"} Jul 14 19:11:16.640904 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:16.640867 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:11:16.640976 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:16.640940 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:11:16.641077 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:16.641060 2576 scope.go:117] "RemoveContainer" containerID="2cc870231f21ce1cfa2ef3c98be3ec04c225ba75548fa803f538db34ab03a877" Jul 14 19:11:16.642957 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:16.642935 2576 generic.go:356] "Generic (PLEG): container finished" podID="2d475a4f-8acf-4cc1-b605-be9c1be74d34" containerID="db810a0d692d52cf730db1d13bd901c90a1945da40c166d19d4cc1fbda8e0cdc" exitCode=0 Jul 14 19:11:16.643097 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:16.642975 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f7zqk" event={"ID":"2d475a4f-8acf-4cc1-b605-be9c1be74d34","Type":"ContainerDied","Data":"db810a0d692d52cf730db1d13bd901c90a1945da40c166d19d4cc1fbda8e0cdc"} Jul 14 19:11:16.658824 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:16.658803 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:11:17.509444 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:17.509417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:17.509629 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:17.509484 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:11:17.509629 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:17.509501 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:11:17.509629 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:17.509601 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:11:17.509735 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:17.509649 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:17.509735 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:17.509708 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:11:17.649028 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:17.648977 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:11:17.649402 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:17.649373 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" event={"ID":"dd9dbd61-c491-4f7a-81b7-1769a9e2cb02","Type":"ContainerStarted","Data":"5268cf6f2d112c2e7c23befd4eca5ca05737fb738dd4114463ee489bee2b355f"} Jul 14 19:11:17.649743 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:17.649692 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:11:17.664925 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:17.664905 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:11:17.677121 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:17.677074 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" podStartSLOduration=9.993274762 podStartE2EDuration="30.677054635s" totalImagesPullingTime="20.683779873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:17.675162798 +0000 UTC m=+30.788412669" watchObservedRunningTime="2026-07-14 19:11:17.677054635 +0000 UTC m=+30.790304506" Jul 14 19:11:18.106735 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:18.106706 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bv2wz"] Jul 14 19:11:18.106867 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:18.106816 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:18.106915 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:18.106898 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:11:18.109676 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:18.109648 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xczln"] Jul 14 19:11:18.109795 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:18.109772 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:11:18.109878 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:18.109859 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:11:18.110440 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:18.110419 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tgldx"] Jul 14 19:11:18.110536 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:18.110501 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:18.110625 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:18.110604 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:11:18.654619 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:18.654566 2576 generic.go:356] "Generic (PLEG): container finished" podID="2d475a4f-8acf-4cc1-b605-be9c1be74d34" containerID="02a7cf4d4811143b89c259cd3b70743c8190585e7d217532e9e43cb5cbb5d7ce" exitCode=0 Jul 14 19:11:18.655006 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:18.654652 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f7zqk" event={"ID":"2d475a4f-8acf-4cc1-b605-be9c1be74d34","Type":"ContainerDied","Data":"02a7cf4d4811143b89c259cd3b70743c8190585e7d217532e9e43cb5cbb5d7ce"} Jul 14 19:11:19.509380 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:19.509341 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:19.509552 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:19.509485 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:11:19.509552 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:19.509541 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:19.509714 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:19.509694 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:11:20.509480 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:20.509448 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:11:20.510054 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:20.509606 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:11:21.095239 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:21.095206 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs\") pod \"network-metrics-daemon-xczln\" (UID: \"8d1706ff-a844-42e9-b6d8-628495d88055\") " pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:11:21.095472 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:21.095370 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:11:21.095472 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:21.095444 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs podName:8d1706ff-a844-42e9-b6d8-628495d88055 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:53.095424887 +0000 UTC m=+66.208674750 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs") pod "network-metrics-daemon-xczln" (UID: "8d1706ff-a844-42e9-b6d8-628495d88055") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:11:21.196489 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:21.196449 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret\") pod \"global-pull-secret-syncer-bv2wz\" (UID: \"5e6c415f-305f-477e-b939-1c2b84a3d770\") " pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:21.196676 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:21.196594 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:11:21.196676 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:21.196663 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret podName:5e6c415f-305f-477e-b939-1c2b84a3d770 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:53.19664374 +0000 UTC m=+66.309893601 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret") pod "global-pull-secret-syncer-bv2wz" (UID: "5e6c415f-305f-477e-b939-1c2b84a3d770") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:11:21.296810 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:21.296768 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwg6s\" (UniqueName: \"kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s\") pod \"network-check-target-tgldx\" (UID: \"67bb23c7-9bee-4350-a4bf-0da1589fbce0\") " pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:21.297008 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:21.296967 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:11:21.297008 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:21.296992 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:11:21.297008 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:21.297002 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cwg6s for pod openshift-network-diagnostics/network-check-target-tgldx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:11:21.297161 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:21.297066 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s podName:67bb23c7-9bee-4350-a4bf-0da1589fbce0 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:53.297049521 +0000 UTC m=+66.410299372 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cwg6s" (UniqueName: "kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s") pod "network-check-target-tgldx" (UID: "67bb23c7-9bee-4350-a4bf-0da1589fbce0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:11:21.509684 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:21.509551 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:21.509684 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:21.509611 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:21.510287 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:21.509706 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:11:21.510287 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:21.509852 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:11:22.508838 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:22.508804 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:11:22.508996 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:22.508921 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:11:23.509313 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:23.509275 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:23.509887 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:23.509395 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bv2wz" podUID="5e6c415f-305f-477e-b939-1c2b84a3d770" Jul 14 19:11:23.509887 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:23.509448 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:23.509887 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:23.509531 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgldx" podUID="67bb23c7-9bee-4350-a4bf-0da1589fbce0" Jul 14 19:11:23.668013 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:23.667976 2576 generic.go:356] "Generic (PLEG): container finished" podID="2d475a4f-8acf-4cc1-b605-be9c1be74d34" containerID="2897faf506c1be1ef40960b5884abdb1aaffee6fc1767d982260b2c13199ac7b" exitCode=0 Jul 14 19:11:23.668169 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:23.668031 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f7zqk" event={"ID":"2d475a4f-8acf-4cc1-b605-be9c1be74d34","Type":"ContainerDied","Data":"2897faf506c1be1ef40960b5884abdb1aaffee6fc1767d982260b2c13199ac7b"} Jul 14 19:11:24.171603 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.171558 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-74.ec2.internal" event="NodeReady" Jul 14 19:11:24.171738 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.171714 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 19:11:24.222525 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.222492 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-29wvt"] Jul 14 19:11:24.224538 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.224518 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-29wvt" Jul 14 19:11:24.226097 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.226069 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j7zhz"] Jul 14 19:11:24.227618 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.227594 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 19:11:24.227722 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.227627 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 19:11:24.227722 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.227601 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h5x4l\"" Jul 14 19:11:24.227923 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.227907 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j7zhz" Jul 14 19:11:24.232090 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.232063 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 19:11:24.232191 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.232145 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cxj8f\"" Jul 14 19:11:24.232191 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.232168 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 19:11:24.232342 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.232328 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 19:11:24.238878 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.238827 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j7zhz"] Jul 14 19:11:24.239429 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.239408 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-29wvt"] Jul 14 19:11:24.318297 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.318260 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3ea0e40-5a86-4c4b-980b-2007a1290795-tmp-dir\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:11:24.318297 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.318311 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgrgv\" (UniqueName: \"kubernetes.io/projected/f3ea0e40-5a86-4c4b-980b-2007a1290795-kube-api-access-kgrgv\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:11:24.318546 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.318352 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e789e3e-9025-425e-bdfb-b8caedfa2767-cert\") pod \"ingress-canary-j7zhz\" (UID: \"5e789e3e-9025-425e-bdfb-b8caedfa2767\") " pod="openshift-ingress-canary/ingress-canary-j7zhz" Jul 14 19:11:24.318546 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.318368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:11:24.318546 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.318385 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5rnb\" (UniqueName: \"kubernetes.io/projected/5e789e3e-9025-425e-bdfb-b8caedfa2767-kube-api-access-g5rnb\") pod \"ingress-canary-j7zhz\" (UID: \"5e789e3e-9025-425e-bdfb-b8caedfa2767\") " pod="openshift-ingress-canary/ingress-canary-j7zhz" Jul 14 19:11:24.318546 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.318498 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f3ea0e40-5a86-4c4b-980b-2007a1290795-config-volume\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:11:24.419710 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.419665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3ea0e40-5a86-4c4b-980b-2007a1290795-tmp-dir\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:11:24.419710 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.419717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgrgv\" (UniqueName: \"kubernetes.io/projected/f3ea0e40-5a86-4c4b-980b-2007a1290795-kube-api-access-kgrgv\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:11:24.419884 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.419835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e789e3e-9025-425e-bdfb-b8caedfa2767-cert\") pod \"ingress-canary-j7zhz\" (UID: \"5e789e3e-9025-425e-bdfb-b8caedfa2767\") " pod="openshift-ingress-canary/ingress-canary-j7zhz" Jul 14 19:11:24.419884 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.419868 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:11:24.419988 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.419887 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5rnb\" (UniqueName: \"kubernetes.io/projected/5e789e3e-9025-425e-bdfb-b8caedfa2767-kube-api-access-g5rnb\") pod \"ingress-canary-j7zhz\" (UID: \"5e789e3e-9025-425e-bdfb-b8caedfa2767\") " pod="openshift-ingress-canary/ingress-canary-j7zhz" Jul 14 19:11:24.419988 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.419945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f3ea0e40-5a86-4c4b-980b-2007a1290795-config-volume\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:11:24.419988 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:24.419980 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:11:24.420120 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:24.420044 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls podName:f3ea0e40-5a86-4c4b-980b-2007a1290795 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:24.920024423 +0000 UTC m=+38.033274281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls") pod "dns-default-29wvt" (UID: "f3ea0e40-5a86-4c4b-980b-2007a1290795") : secret "dns-default-metrics-tls" not found Jul 14 19:11:24.420120 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.420061 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3ea0e40-5a86-4c4b-980b-2007a1290795-tmp-dir\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:11:24.420436 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.420419 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f3ea0e40-5a86-4c4b-980b-2007a1290795-config-volume\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:11:24.423935 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.423891 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e789e3e-9025-425e-bdfb-b8caedfa2767-cert\") pod \"ingress-canary-j7zhz\" (UID: \"5e789e3e-9025-425e-bdfb-b8caedfa2767\") " pod="openshift-ingress-canary/ingress-canary-j7zhz" Jul 14 19:11:24.428755 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.428724 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgrgv\" (UniqueName: \"kubernetes.io/projected/f3ea0e40-5a86-4c4b-980b-2007a1290795-kube-api-access-kgrgv\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:11:24.428866 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.428838 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5rnb\" (UniqueName: \"kubernetes.io/projected/5e789e3e-9025-425e-bdfb-b8caedfa2767-kube-api-access-g5rnb\") pod \"ingress-canary-j7zhz\" (UID: \"5e789e3e-9025-425e-bdfb-b8caedfa2767\") " pod="openshift-ingress-canary/ingress-canary-j7zhz" Jul 14 19:11:24.509025 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.508987 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:11:24.511753 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.511735 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 19:11:24.512074 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.511855 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xxpfz\"" Jul 14 19:11:24.544398 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.544357 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j7zhz" Jul 14 19:11:24.673772 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.673596 2576 generic.go:356] "Generic (PLEG): container finished" podID="2d475a4f-8acf-4cc1-b605-be9c1be74d34" containerID="82b72c349db030d2ddc495afe1c7c7a375c80e629555a63b844517cdea48897a" exitCode=0 Jul 14 19:11:24.673772 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.673672 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f7zqk" event={"ID":"2d475a4f-8acf-4cc1-b605-be9c1be74d34","Type":"ContainerDied","Data":"82b72c349db030d2ddc495afe1c7c7a375c80e629555a63b844517cdea48897a"} Jul 14 19:11:24.677635 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.677609 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j7zhz"] Jul 14 19:11:24.682336 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:11:24.682310 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e789e3e_9025_425e_bdfb_b8caedfa2767.slice/crio-847a570690d6da93860a7d6b24d45f6f1697332cfe52ea2f91ef2eeb7124871e WatchSource:0}: Error finding container 847a570690d6da93860a7d6b24d45f6f1697332cfe52ea2f91ef2eeb7124871e: Status 404 returned error can't find the container with id 847a570690d6da93860a7d6b24d45f6f1697332cfe52ea2f91ef2eeb7124871e Jul 14 19:11:24.924263 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:24.924222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:11:24.924462 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:24.924370 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:11:24.924462 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:24.924434 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls podName:f3ea0e40-5a86-4c4b-980b-2007a1290795 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:25.924419115 +0000 UTC m=+39.037668963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls") pod "dns-default-29wvt" (UID: "f3ea0e40-5a86-4c4b-980b-2007a1290795") : secret "dns-default-metrics-tls" not found Jul 14 19:11:25.509411 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:25.509376 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:25.509629 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:25.509576 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:25.512598 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:25.512569 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 19:11:25.513039 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:25.512620 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 19:11:25.513039 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:25.512639 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 19:11:25.513744 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:25.513727 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jwrqs\"" Jul 14 19:11:25.678315 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:25.678270 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j7zhz" event={"ID":"5e789e3e-9025-425e-bdfb-b8caedfa2767","Type":"ContainerStarted","Data":"847a570690d6da93860a7d6b24d45f6f1697332cfe52ea2f91ef2eeb7124871e"} Jul 14 19:11:25.681949 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:25.681924 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f7zqk" event={"ID":"2d475a4f-8acf-4cc1-b605-be9c1be74d34","Type":"ContainerStarted","Data":"3425498811f69c3b68ed26de60ce39a5ce86f7dccfa27884fadaec4c68bf357b"} Jul 14 19:11:25.707866 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:25.705780 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-f7zqk" podStartSLOduration=24.028634026 podStartE2EDuration="37.705761043s" totalImagesPullingTime="13.677127017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:25.70528283 +0000 UTC m=+38.818532702" watchObservedRunningTime="2026-07-14 19:11:25.705761043 +0000 UTC m=+38.819010914" Jul 14 19:11:25.933643 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:25.933542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:11:25.933798 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:25.933711 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:11:25.933798 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:25.933790 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls podName:f3ea0e40-5a86-4c4b-980b-2007a1290795 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:27.933770152 +0000 UTC m=+41.047020000 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls") pod "dns-default-29wvt" (UID: "f3ea0e40-5a86-4c4b-980b-2007a1290795") : secret "dns-default-metrics-tls" not found Jul 14 19:11:27.689272 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:27.689041 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j7zhz" event={"ID":"5e789e3e-9025-425e-bdfb-b8caedfa2767","Type":"ContainerStarted","Data":"86d080f9d4e91bb8083e1a1f1ebf9cf9752313a830b419d371cd69ddf502f800"} Jul 14 19:11:27.708794 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:27.708741 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j7zhz" podStartSLOduration=1.215082503 podStartE2EDuration="3.708723408s" totalImagesPullingTime="2.493640905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:27.708509376 +0000 UTC m=+40.821759246" watchObservedRunningTime="2026-07-14 19:11:27.708723408 +0000 UTC m=+40.821973279" Jul 14 19:11:27.948535 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:27.948438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:11:27.948715 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:27.948603 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:11:27.948715 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:27.948665 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls podName:f3ea0e40-5a86-4c4b-980b-2007a1290795 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:31.948649816 +0000 UTC m=+45.061899664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls") pod "dns-default-29wvt" (UID: "f3ea0e40-5a86-4c4b-980b-2007a1290795") : secret "dns-default-metrics-tls" not found Jul 14 19:11:31.979887 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:31.979838 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:11:31.980248 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:31.979971 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:11:31.980248 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:31.980025 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls podName:f3ea0e40-5a86-4c4b-980b-2007a1290795 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:39.980011503 +0000 UTC m=+53.093261351 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls") pod "dns-default-29wvt" (UID: "f3ea0e40-5a86-4c4b-980b-2007a1290795") : secret "dns-default-metrics-tls" not found Jul 14 19:11:40.035607 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:40.035537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:11:40.036160 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:40.035698 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:11:40.036160 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:40.035771 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls podName:f3ea0e40-5a86-4c4b-980b-2007a1290795 nodeName:}" failed. No retries permitted until 2026-07-14 19:11:56.035755715 +0000 UTC m=+69.149005563 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls") pod "dns-default-29wvt" (UID: "f3ea0e40-5a86-4c4b-980b-2007a1290795") : secret "dns-default-metrics-tls" not found Jul 14 19:11:49.667492 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:49.667466 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tg2x9" Jul 14 19:11:53.125154 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:53.125107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs\") pod \"network-metrics-daemon-xczln\" (UID: \"8d1706ff-a844-42e9-b6d8-628495d88055\") " pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:11:53.128056 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:53.128037 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 19:11:53.135428 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:53.135410 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 14 19:11:53.135486 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:53.135468 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs podName:8d1706ff-a844-42e9-b6d8-628495d88055 nodeName:}" failed. No retries permitted until 2026-07-14 19:12:57.135452538 +0000 UTC m=+130.248702390 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs") pod "network-metrics-daemon-xczln" (UID: "8d1706ff-a844-42e9-b6d8-628495d88055") : secret "metrics-daemon-secret" not found Jul 14 19:11:53.225464 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:53.225440 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret\") pod \"global-pull-secret-syncer-bv2wz\" (UID: \"5e6c415f-305f-477e-b939-1c2b84a3d770\") " pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:53.228516 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:53.228499 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 19:11:53.239678 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:53.239655 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e6c415f-305f-477e-b939-1c2b84a3d770-original-pull-secret\") pod \"global-pull-secret-syncer-bv2wz\" (UID: \"5e6c415f-305f-477e-b939-1c2b84a3d770\") " pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:53.326318 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:53.326277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwg6s\" (UniqueName: \"kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s\") pod \"network-check-target-tgldx\" (UID: \"67bb23c7-9bee-4350-a4bf-0da1589fbce0\") " pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:53.329185 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:53.329167 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 19:11:53.339501 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:53.339479 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 19:11:53.350027 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:53.349998 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwg6s\" (UniqueName: \"kubernetes.io/projected/67bb23c7-9bee-4350-a4bf-0da1589fbce0-kube-api-access-cwg6s\") pod \"network-check-target-tgldx\" (UID: \"67bb23c7-9bee-4350-a4bf-0da1589fbce0\") " pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:53.420916 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:53.420862 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bv2wz" Jul 14 19:11:53.430350 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:53.430331 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jwrqs\"" Jul 14 19:11:53.437673 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:53.437650 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:53.561420 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:53.561387 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bv2wz"] Jul 14 19:11:53.564831 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:11:53.564801 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6c415f_305f_477e_b939_1c2b84a3d770.slice/crio-344548da33a505164f13ecf2dcb5ea850be3d12a4d2f7c93f0401b6a78ced796 WatchSource:0}: Error finding container 344548da33a505164f13ecf2dcb5ea850be3d12a4d2f7c93f0401b6a78ced796: Status 404 returned error can't find the container with id 344548da33a505164f13ecf2dcb5ea850be3d12a4d2f7c93f0401b6a78ced796 Jul 14 19:11:53.584647 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:53.584629 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tgldx"] Jul 14 19:11:53.586525 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:11:53.586506 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67bb23c7_9bee_4350_a4bf_0da1589fbce0.slice/crio-2a6179fafda21f0879bfab949f7f3dfa1a192ddafe1f766fe4a4b9626d789953 WatchSource:0}: Error finding container 2a6179fafda21f0879bfab949f7f3dfa1a192ddafe1f766fe4a4b9626d789953: Status 404 returned error can't find the container with id 2a6179fafda21f0879bfab949f7f3dfa1a192ddafe1f766fe4a4b9626d789953 Jul 14 19:11:53.748526 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:53.748435 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bv2wz" event={"ID":"5e6c415f-305f-477e-b939-1c2b84a3d770","Type":"ContainerStarted","Data":"344548da33a505164f13ecf2dcb5ea850be3d12a4d2f7c93f0401b6a78ced796"} Jul 14 19:11:53.749452 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:53.749432 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tgldx" event={"ID":"67bb23c7-9bee-4350-a4bf-0da1589fbce0","Type":"ContainerStarted","Data":"2a6179fafda21f0879bfab949f7f3dfa1a192ddafe1f766fe4a4b9626d789953"} Jul 14 19:11:56.048844 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:56.048802 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:11:56.049329 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:56.048969 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:11:56.049329 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:11:56.049039 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls podName:f3ea0e40-5a86-4c4b-980b-2007a1290795 nodeName:}" failed. No retries permitted until 2026-07-14 19:12:28.049019555 +0000 UTC m=+101.162269411 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls") pod "dns-default-29wvt" (UID: "f3ea0e40-5a86-4c4b-980b-2007a1290795") : secret "dns-default-metrics-tls" not found Jul 14 19:11:58.764287 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:58.764246 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tgldx" event={"ID":"67bb23c7-9bee-4350-a4bf-0da1589fbce0","Type":"ContainerStarted","Data":"a0ca4bb72d692675b0d4671437f426fcb1d8a257ed61cc66afc2560ab12e4765"} Jul 14 19:11:58.764733 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:58.764365 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:11:58.765719 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:58.765697 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bv2wz" event={"ID":"5e6c415f-305f-477e-b939-1c2b84a3d770","Type":"ContainerStarted","Data":"0a28b9df1686ff8583dcebeb725e2b3d00eb750c6eca926fc3a21ad4fcc9d147"} Jul 14 19:11:58.781806 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:58.781759 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tgldx" podStartSLOduration=67.741732756 podStartE2EDuration="1m10.781745694s" totalImagesPullingTime="3.040012938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:58.780438162 +0000 UTC m=+71.893688054" watchObservedRunningTime="2026-07-14 19:11:58.781745694 +0000 UTC m=+71.894995563" Jul 14 19:11:58.795294 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:11:58.795250 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bv2wz" podStartSLOduration=66.335282214 podStartE2EDuration="1m10.795233538s" totalImagesPullingTime="4.459951324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:11:58.795054319 +0000 UTC m=+71.908304193" watchObservedRunningTime="2026-07-14 19:11:58.795233538 +0000 UTC m=+71.908483409" Jul 14 19:12:28.057181 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:12:28.057142 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:12:28.057755 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:12:28.057336 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:12:28.057755 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:12:28.057413 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls podName:f3ea0e40-5a86-4c4b-980b-2007a1290795 nodeName:}" failed. No retries permitted until 2026-07-14 19:13:32.057393772 +0000 UTC m=+165.170643620 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls") pod "dns-default-29wvt" (UID: "f3ea0e40-5a86-4c4b-980b-2007a1290795") : secret "dns-default-metrics-tls" not found Jul 14 19:12:29.770596 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:12:29.770547 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tgldx" Jul 14 19:12:57.156975 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:12:57.156931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs\") pod \"network-metrics-daemon-xczln\" (UID: \"8d1706ff-a844-42e9-b6d8-628495d88055\") " pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:12:57.157440 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:12:57.157079 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 14 19:12:57.157440 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:12:57.157145 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs podName:8d1706ff-a844-42e9-b6d8-628495d88055 nodeName:}" failed. No retries permitted until 2026-07-14 19:14:59.15713017 +0000 UTC m=+252.270380021 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs") pod "network-metrics-daemon-xczln" (UID: "8d1706ff-a844-42e9-b6d8-628495d88055") : secret "metrics-daemon-secret" not found Jul 14 19:13:27.237479 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:13:27.237433 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-29wvt" podUID="f3ea0e40-5a86-4c4b-980b-2007a1290795" Jul 14 19:13:27.517254 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:13:27.517174 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:13:27.977828 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:13:27.977796 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-29wvt" Jul 14 19:13:32.096537 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:13:32.096494 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:13:32.097013 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:13:32.096638 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:13:32.097013 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:13:32.096710 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls podName:f3ea0e40-5a86-4c4b-980b-2007a1290795 nodeName:}" failed. No retries permitted until 2026-07-14 19:15:34.096696962 +0000 UTC m=+287.209946811 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls") pod "dns-default-29wvt" (UID: "f3ea0e40-5a86-4c4b-980b-2007a1290795") : secret "dns-default-metrics-tls" not found Jul 14 19:13:42.509121 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:13:42.509027 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:14:51.479363 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.479324 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-tt2st"] Jul 14 19:14:51.481425 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.481406 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-pxckb"] Jul 14 19:14:51.481543 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.481485 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-tt2st" Jul 14 19:14:51.483318 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.483302 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.485507 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.485491 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6g6wd\"" Jul 14 19:14:51.486230 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.486215 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 14 19:14:51.486283 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.486219 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 19:14:51.486975 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.486953 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-d87wq\"" Jul 14 19:14:51.487251 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.487236 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 14 19:14:51.487330 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.487250 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 19:14:51.498763 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.498740 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 14 19:14:51.499312 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.499294 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-tt2st"] Jul 14 19:14:51.504011 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.503987 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-pxckb"] Jul 14 19:14:51.532066 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.532033 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjsd5\" (UniqueName: \"kubernetes.io/projected/e92e640a-57c7-4a12-89cb-701c6f9bf3e6-kube-api-access-wjsd5\") pod \"insights-operator-68d9dd94cf-pxckb\" (UID: \"e92e640a-57c7-4a12-89cb-701c6f9bf3e6\") " pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.532066 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.532068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92e640a-57c7-4a12-89cb-701c6f9bf3e6-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-pxckb\" (UID: \"e92e640a-57c7-4a12-89cb-701c6f9bf3e6\") " pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.532314 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.532092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e92e640a-57c7-4a12-89cb-701c6f9bf3e6-snapshots\") pod \"insights-operator-68d9dd94cf-pxckb\" (UID: \"e92e640a-57c7-4a12-89cb-701c6f9bf3e6\") " pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.532314 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.532119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e92e640a-57c7-4a12-89cb-701c6f9bf3e6-serving-cert\") pod \"insights-operator-68d9dd94cf-pxckb\" (UID: \"e92e640a-57c7-4a12-89cb-701c6f9bf3e6\") " pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.532314 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.532244 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e92e640a-57c7-4a12-89cb-701c6f9bf3e6-tmp\") pod \"insights-operator-68d9dd94cf-pxckb\" (UID: \"e92e640a-57c7-4a12-89cb-701c6f9bf3e6\") " pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.532314 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.532285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-945nj\" (UniqueName: \"kubernetes.io/projected/63ff1a4c-4415-486c-a331-e5a37b4cbaaa-kube-api-access-945nj\") pod \"network-check-source-869dcfcdcd-tt2st\" (UID: \"63ff1a4c-4415-486c-a331-e5a37b4cbaaa\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-tt2st" Jul 14 19:14:51.532444 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.532342 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92e640a-57c7-4a12-89cb-701c6f9bf3e6-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-pxckb\" (UID: \"e92e640a-57c7-4a12-89cb-701c6f9bf3e6\") " pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.569283 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.569255 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b"] Jul 14 19:14:51.571246 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.571230 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b" Jul 14 19:14:51.574247 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.574217 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 14 19:14:51.574507 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.574487 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 14 19:14:51.574615 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.574578 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 14 19:14:51.574615 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.574612 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nrzrg\"" Jul 14 19:14:51.574782 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.574594 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:14:51.576393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.576372 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-79c9798784-bdl9b"] Jul 14 19:14:51.578354 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.578191 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd"] Jul 14 19:14:51.578430 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.578388 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:51.580438 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.580422 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd"] Jul 14 19:14:51.580529 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.580516 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd" Jul 14 19:14:51.581153 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.581137 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 19:14:51.581363 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.581341 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 14 19:14:51.581440 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.581370 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 14 19:14:51.581440 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.581346 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-82l7q\"" Jul 14 19:14:51.581557 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.581508 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 14 19:14:51.581639 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.581594 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 14 19:14:51.581639 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.581633 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 19:14:51.582393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.582376 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" Jul 14 19:14:51.585002 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.584987 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 14 19:14:51.586332 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.586317 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 14 19:14:51.586517 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.586503 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 14 19:14:51.586517 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.586513 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 14 19:14:51.586763 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.586507 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:14:51.586832 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.586796 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b"] Jul 14 19:14:51.588412 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.588395 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 14 19:14:51.588906 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.588886 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 19:14:51.588906 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.588901 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 19:14:51.589057 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.588924 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-f6d5d\"" Jul 14 19:14:51.589057 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.588886 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-nc49p\"" Jul 14 19:14:51.595082 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.595058 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd"] Jul 14 19:14:51.595979 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.595956 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd"] Jul 14 19:14:51.598764 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.598745 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79c9798784-bdl9b"] Jul 14 19:14:51.633303 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.633272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-default-certificate\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:51.633303 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.633311 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c048bba9-c26d-4398-a1ce-9b4a9be0ab89-config\") pod \"service-ca-operator-7695fc6c8c-rq5xd\" (UID: \"c048bba9-c26d-4398-a1ce-9b4a9be0ab89\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd" Jul 14 19:14:51.633569 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.633332 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b20459ce-1f34-4dcd-becb-522fee08d273-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-dnr7b\" (UID: \"b20459ce-1f34-4dcd-becb-522fee08d273\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b" Jul 14 19:14:51.633569 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.633353 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzk8h\" (UniqueName: \"kubernetes.io/projected/01ac54b9-9511-4e3c-afcd-482292702a0b-kube-api-access-wzk8h\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:51.633569 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.633472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e92e640a-57c7-4a12-89cb-701c6f9bf3e6-tmp\") pod \"insights-operator-68d9dd94cf-pxckb\" (UID: \"e92e640a-57c7-4a12-89cb-701c6f9bf3e6\") " pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.633569 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.633508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmtnz\" (UniqueName: \"kubernetes.io/projected/c90a7de7-c072-40bd-bebc-ee4996a2e763-kube-api-access-qmtnz\") pod \"cluster-monitoring-operator-5f7f494dd4-4mvrd\" (UID: \"c90a7de7-c072-40bd-bebc-ee4996a2e763\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" Jul 14 19:14:51.633569 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.633540 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b20459ce-1f34-4dcd-becb-522fee08d273-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-dnr7b\" (UID: \"b20459ce-1f34-4dcd-becb-522fee08d273\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b" Jul 14 19:14:51.633871 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.633577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsd5\" (UniqueName: \"kubernetes.io/projected/e92e640a-57c7-4a12-89cb-701c6f9bf3e6-kube-api-access-wjsd5\") pod \"insights-operator-68d9dd94cf-pxckb\" (UID: \"e92e640a-57c7-4a12-89cb-701c6f9bf3e6\") " pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.633871 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.633630 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxsqq\" (UniqueName: \"kubernetes.io/projected/c048bba9-c26d-4398-a1ce-9b4a9be0ab89-kube-api-access-gxsqq\") pod \"service-ca-operator-7695fc6c8c-rq5xd\" (UID: \"c048bba9-c26d-4398-a1ce-9b4a9be0ab89\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd" Jul 14 19:14:51.633871 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.633658 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92e640a-57c7-4a12-89cb-701c6f9bf3e6-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-pxckb\" (UID: \"e92e640a-57c7-4a12-89cb-701c6f9bf3e6\") " pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.633871 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.633684 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84nq6\" (UniqueName: \"kubernetes.io/projected/b20459ce-1f34-4dcd-becb-522fee08d273-kube-api-access-84nq6\") pod \"kube-storage-version-migrator-operator-799cb7f958-dnr7b\" (UID: \"b20459ce-1f34-4dcd-becb-522fee08d273\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b" Jul 14 19:14:51.633871 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.633730 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-stats-auth\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:51.633871 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.633758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92e640a-57c7-4a12-89cb-701c6f9bf3e6-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-pxckb\" (UID: \"e92e640a-57c7-4a12-89cb-701c6f9bf3e6\") " pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.633871 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.633781 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e92e640a-57c7-4a12-89cb-701c6f9bf3e6-serving-cert\") pod \"insights-operator-68d9dd94cf-pxckb\" (UID: \"e92e640a-57c7-4a12-89cb-701c6f9bf3e6\") " pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.633871 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.633809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e92e640a-57c7-4a12-89cb-701c6f9bf3e6-snapshots\") pod \"insights-operator-68d9dd94cf-pxckb\" (UID: \"e92e640a-57c7-4a12-89cb-701c6f9bf3e6\") " pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.634274 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.633924 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e92e640a-57c7-4a12-89cb-701c6f9bf3e6-tmp\") pod \"insights-operator-68d9dd94cf-pxckb\" (UID: \"e92e640a-57c7-4a12-89cb-701c6f9bf3e6\") " pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.634274 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.633925 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ac54b9-9511-4e3c-afcd-482292702a0b-service-ca-bundle\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:51.634274 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.633978 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c90a7de7-c072-40bd-bebc-ee4996a2e763-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-4mvrd\" (UID: \"c90a7de7-c072-40bd-bebc-ee4996a2e763\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" Jul 14 19:14:51.634274 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.634009 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-metrics-certs\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:51.634274 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.634091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-945nj\" (UniqueName: \"kubernetes.io/projected/63ff1a4c-4415-486c-a331-e5a37b4cbaaa-kube-api-access-945nj\") pod \"network-check-source-869dcfcdcd-tt2st\" (UID: \"63ff1a4c-4415-486c-a331-e5a37b4cbaaa\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-tt2st" Jul 14 19:14:51.634274 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.634133 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c048bba9-c26d-4398-a1ce-9b4a9be0ab89-serving-cert\") pod \"service-ca-operator-7695fc6c8c-rq5xd\" (UID: \"c048bba9-c26d-4398-a1ce-9b4a9be0ab89\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd" Jul 14 19:14:51.634274 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.634181 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c90a7de7-c072-40bd-bebc-ee4996a2e763-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-4mvrd\" (UID: \"c90a7de7-c072-40bd-bebc-ee4996a2e763\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" Jul 14 19:14:51.634274 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.634243 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92e640a-57c7-4a12-89cb-701c6f9bf3e6-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-pxckb\" (UID: \"e92e640a-57c7-4a12-89cb-701c6f9bf3e6\") " pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.634607 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.634423 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e92e640a-57c7-4a12-89cb-701c6f9bf3e6-snapshots\") pod \"insights-operator-68d9dd94cf-pxckb\" (UID: \"e92e640a-57c7-4a12-89cb-701c6f9bf3e6\") " pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.634664 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.634648 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92e640a-57c7-4a12-89cb-701c6f9bf3e6-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-pxckb\" (UID: \"e92e640a-57c7-4a12-89cb-701c6f9bf3e6\") " pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.636131 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.636116 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e92e640a-57c7-4a12-89cb-701c6f9bf3e6-serving-cert\") pod \"insights-operator-68d9dd94cf-pxckb\" (UID: \"e92e640a-57c7-4a12-89cb-701c6f9bf3e6\") " pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.645757 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.645735 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjsd5\" (UniqueName: \"kubernetes.io/projected/e92e640a-57c7-4a12-89cb-701c6f9bf3e6-kube-api-access-wjsd5\") pod \"insights-operator-68d9dd94cf-pxckb\" (UID: \"e92e640a-57c7-4a12-89cb-701c6f9bf3e6\") " pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.645914 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.645898 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-945nj\" (UniqueName: \"kubernetes.io/projected/63ff1a4c-4415-486c-a331-e5a37b4cbaaa-kube-api-access-945nj\") pod \"network-check-source-869dcfcdcd-tt2st\" (UID: \"63ff1a4c-4415-486c-a331-e5a37b4cbaaa\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-tt2st" Jul 14 19:14:51.735145 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.735057 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-default-certificate\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:51.735145 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.735092 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c048bba9-c26d-4398-a1ce-9b4a9be0ab89-config\") pod \"service-ca-operator-7695fc6c8c-rq5xd\" (UID: \"c048bba9-c26d-4398-a1ce-9b4a9be0ab89\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd" Jul 14 19:14:51.735145 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.735113 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b20459ce-1f34-4dcd-becb-522fee08d273-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-dnr7b\" (UID: \"b20459ce-1f34-4dcd-becb-522fee08d273\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b" Jul 14 19:14:51.735409 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.735145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzk8h\" (UniqueName: \"kubernetes.io/projected/01ac54b9-9511-4e3c-afcd-482292702a0b-kube-api-access-wzk8h\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:51.735409 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.735332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmtnz\" (UniqueName: \"kubernetes.io/projected/c90a7de7-c072-40bd-bebc-ee4996a2e763-kube-api-access-qmtnz\") pod \"cluster-monitoring-operator-5f7f494dd4-4mvrd\" (UID: \"c90a7de7-c072-40bd-bebc-ee4996a2e763\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" Jul 14 19:14:51.735409 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.735372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b20459ce-1f34-4dcd-becb-522fee08d273-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-dnr7b\" (UID: \"b20459ce-1f34-4dcd-becb-522fee08d273\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b" Jul 14 19:14:51.735560 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.735411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxsqq\" (UniqueName: \"kubernetes.io/projected/c048bba9-c26d-4398-a1ce-9b4a9be0ab89-kube-api-access-gxsqq\") pod \"service-ca-operator-7695fc6c8c-rq5xd\" (UID: \"c048bba9-c26d-4398-a1ce-9b4a9be0ab89\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd" Jul 14 19:14:51.735645 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.735555 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84nq6\" (UniqueName: \"kubernetes.io/projected/b20459ce-1f34-4dcd-becb-522fee08d273-kube-api-access-84nq6\") pod \"kube-storage-version-migrator-operator-799cb7f958-dnr7b\" (UID: \"b20459ce-1f34-4dcd-becb-522fee08d273\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b" Jul 14 19:14:51.735645 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.735616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-stats-auth\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:51.735744 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.735659 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ac54b9-9511-4e3c-afcd-482292702a0b-service-ca-bundle\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:51.735744 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.735699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c90a7de7-c072-40bd-bebc-ee4996a2e763-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-4mvrd\" (UID: \"c90a7de7-c072-40bd-bebc-ee4996a2e763\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" Jul 14 19:14:51.735744 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.735731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-metrics-certs\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:51.735884 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.735754 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c048bba9-c26d-4398-a1ce-9b4a9be0ab89-config\") pod \"service-ca-operator-7695fc6c8c-rq5xd\" (UID: \"c048bba9-c26d-4398-a1ce-9b4a9be0ab89\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd" Jul 14 19:14:51.735884 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.735762 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b20459ce-1f34-4dcd-becb-522fee08d273-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-dnr7b\" (UID: \"b20459ce-1f34-4dcd-becb-522fee08d273\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b" Jul 14 19:14:51.735884 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.735772 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c048bba9-c26d-4398-a1ce-9b4a9be0ab89-serving-cert\") pod \"service-ca-operator-7695fc6c8c-rq5xd\" (UID: \"c048bba9-c26d-4398-a1ce-9b4a9be0ab89\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd" Jul 14 19:14:51.735884 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:51.735852 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 19:14:51.736069 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:51.735852 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 19:14:51.736069 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:51.735857 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/01ac54b9-9511-4e3c-afcd-482292702a0b-service-ca-bundle podName:01ac54b9-9511-4e3c-afcd-482292702a0b nodeName:}" failed. No retries permitted until 2026-07-14 19:14:52.235835689 +0000 UTC m=+245.349085540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/01ac54b9-9511-4e3c-afcd-482292702a0b-service-ca-bundle") pod "router-default-79c9798784-bdl9b" (UID: "01ac54b9-9511-4e3c-afcd-482292702a0b") : configmap references non-existent config key: service-ca.crt Jul 14 19:14:51.736069 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.735939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c90a7de7-c072-40bd-bebc-ee4996a2e763-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-4mvrd\" (UID: \"c90a7de7-c072-40bd-bebc-ee4996a2e763\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" Jul 14 19:14:51.736069 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:51.735993 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c90a7de7-c072-40bd-bebc-ee4996a2e763-cluster-monitoring-operator-tls podName:c90a7de7-c072-40bd-bebc-ee4996a2e763 nodeName:}" failed. No retries permitted until 2026-07-14 19:14:52.2359775 +0000 UTC m=+245.349227360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c90a7de7-c072-40bd-bebc-ee4996a2e763-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-4mvrd" (UID: "c90a7de7-c072-40bd-bebc-ee4996a2e763") : secret "cluster-monitoring-operator-tls" not found Jul 14 19:14:51.736069 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:51.736009 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-metrics-certs podName:01ac54b9-9511-4e3c-afcd-482292702a0b nodeName:}" failed. No retries permitted until 2026-07-14 19:14:52.235998975 +0000 UTC m=+245.349248824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-metrics-certs") pod "router-default-79c9798784-bdl9b" (UID: "01ac54b9-9511-4e3c-afcd-482292702a0b") : secret "router-metrics-certs-default" not found Jul 14 19:14:51.736606 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.736570 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c90a7de7-c072-40bd-bebc-ee4996a2e763-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-4mvrd\" (UID: \"c90a7de7-c072-40bd-bebc-ee4996a2e763\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" Jul 14 19:14:51.737810 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.737785 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b20459ce-1f34-4dcd-becb-522fee08d273-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-dnr7b\" (UID: \"b20459ce-1f34-4dcd-becb-522fee08d273\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b" Jul 14 19:14:51.737987 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.737972 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-default-certificate\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:51.738087 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.738071 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c048bba9-c26d-4398-a1ce-9b4a9be0ab89-serving-cert\") pod \"service-ca-operator-7695fc6c8c-rq5xd\" (UID: \"c048bba9-c26d-4398-a1ce-9b4a9be0ab89\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd" Jul 14 19:14:51.738341 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.738322 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-stats-auth\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:51.750508 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.750484 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84nq6\" (UniqueName: \"kubernetes.io/projected/b20459ce-1f34-4dcd-becb-522fee08d273-kube-api-access-84nq6\") pod \"kube-storage-version-migrator-operator-799cb7f958-dnr7b\" (UID: \"b20459ce-1f34-4dcd-becb-522fee08d273\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b" Jul 14 19:14:51.750645 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.750562 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzk8h\" (UniqueName: \"kubernetes.io/projected/01ac54b9-9511-4e3c-afcd-482292702a0b-kube-api-access-wzk8h\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:51.751405 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.751383 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmtnz\" (UniqueName: \"kubernetes.io/projected/c90a7de7-c072-40bd-bebc-ee4996a2e763-kube-api-access-qmtnz\") pod \"cluster-monitoring-operator-5f7f494dd4-4mvrd\" (UID: \"c90a7de7-c072-40bd-bebc-ee4996a2e763\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" Jul 14 19:14:51.751505 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.751448 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxsqq\" (UniqueName: \"kubernetes.io/projected/c048bba9-c26d-4398-a1ce-9b4a9be0ab89-kube-api-access-gxsqq\") pod \"service-ca-operator-7695fc6c8c-rq5xd\" (UID: \"c048bba9-c26d-4398-a1ce-9b4a9be0ab89\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd" Jul 14 19:14:51.793471 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.793429 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-tt2st" Jul 14 19:14:51.799644 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.799625 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" Jul 14 19:14:51.882631 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.882599 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b" Jul 14 19:14:51.895861 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.895829 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd" Jul 14 19:14:51.926191 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.925919 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-tt2st"] Jul 14 19:14:51.928884 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:14:51.928846 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ff1a4c_4415_486c_a331_e5a37b4cbaaa.slice/crio-50d612a2c97b87423a56f44dba0f5a90f72af8598a63d12942cc00e3d392ed06 WatchSource:0}: Error finding container 50d612a2c97b87423a56f44dba0f5a90f72af8598a63d12942cc00e3d392ed06: Status 404 returned error can't find the container with id 50d612a2c97b87423a56f44dba0f5a90f72af8598a63d12942cc00e3d392ed06 Jul 14 19:14:51.947047 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:51.947023 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-pxckb"] Jul 14 19:14:51.955787 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:14:51.955499 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92e640a_57c7_4a12_89cb_701c6f9bf3e6.slice/crio-7485c5ef673ccd8c3101244a0f82cf665d861f6bc8e0a5c847dc7e3240e094aa WatchSource:0}: Error finding container 7485c5ef673ccd8c3101244a0f82cf665d861f6bc8e0a5c847dc7e3240e094aa: Status 404 returned error can't find the container with id 7485c5ef673ccd8c3101244a0f82cf665d861f6bc8e0a5c847dc7e3240e094aa Jul 14 19:14:52.026729 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:52.026690 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b"] Jul 14 19:14:52.029761 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:14:52.029724 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20459ce_1f34_4dcd_becb_522fee08d273.slice/crio-10e4adbf82888b6dfe528291aa5f5cdba76eca0a107bae31efd98b9aa6dd100c WatchSource:0}: Error finding container 10e4adbf82888b6dfe528291aa5f5cdba76eca0a107bae31efd98b9aa6dd100c: Status 404 returned error can't find the container with id 10e4adbf82888b6dfe528291aa5f5cdba76eca0a107bae31efd98b9aa6dd100c Jul 14 19:14:52.037289 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:52.037265 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd"] Jul 14 19:14:52.040452 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:14:52.040426 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc048bba9_c26d_4398_a1ce_9b4a9be0ab89.slice/crio-1fd3bdfc166101d8f564a6032097781b4d49f98b30494b089d530a143ad44c11 WatchSource:0}: Error finding container 1fd3bdfc166101d8f564a6032097781b4d49f98b30494b089d530a143ad44c11: Status 404 returned error can't find the container with id 1fd3bdfc166101d8f564a6032097781b4d49f98b30494b089d530a143ad44c11 Jul 14 19:14:52.177666 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:52.177626 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b" event={"ID":"b20459ce-1f34-4dcd-becb-522fee08d273","Type":"ContainerStarted","Data":"10e4adbf82888b6dfe528291aa5f5cdba76eca0a107bae31efd98b9aa6dd100c"} Jul 14 19:14:52.182648 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:52.178880 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" event={"ID":"e92e640a-57c7-4a12-89cb-701c6f9bf3e6","Type":"ContainerStarted","Data":"7485c5ef673ccd8c3101244a0f82cf665d861f6bc8e0a5c847dc7e3240e094aa"} Jul 14 19:14:52.184016 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:52.183985 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd" event={"ID":"c048bba9-c26d-4398-a1ce-9b4a9be0ab89","Type":"ContainerStarted","Data":"1fd3bdfc166101d8f564a6032097781b4d49f98b30494b089d530a143ad44c11"} Jul 14 19:14:52.185355 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:52.185332 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-tt2st" event={"ID":"63ff1a4c-4415-486c-a331-e5a37b4cbaaa","Type":"ContainerStarted","Data":"97233d1088631dd340de6f096c3695313a7e579508fdd80f5bb78ef903e8b7db"} Jul 14 19:14:52.185452 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:52.185372 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-tt2st" event={"ID":"63ff1a4c-4415-486c-a331-e5a37b4cbaaa","Type":"ContainerStarted","Data":"50d612a2c97b87423a56f44dba0f5a90f72af8598a63d12942cc00e3d392ed06"} Jul 14 19:14:52.200864 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:52.200811 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-tt2st" podStartSLOduration=1.200796483 podStartE2EDuration="1.200796483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:14:52.200645277 +0000 UTC m=+245.313895150" watchObservedRunningTime="2026-07-14 19:14:52.200796483 +0000 UTC m=+245.314046354" Jul 14 19:14:52.242179 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:52.242080 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ac54b9-9511-4e3c-afcd-482292702a0b-service-ca-bundle\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:52.242179 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:52.242124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c90a7de7-c072-40bd-bebc-ee4996a2e763-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-4mvrd\" (UID: \"c90a7de7-c072-40bd-bebc-ee4996a2e763\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" Jul 14 19:14:52.242179 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:52.242152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-metrics-certs\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:52.242491 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:52.242255 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 19:14:52.242491 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:52.242271 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 19:14:52.242491 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:52.242302 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/01ac54b9-9511-4e3c-afcd-482292702a0b-service-ca-bundle podName:01ac54b9-9511-4e3c-afcd-482292702a0b nodeName:}" failed. No retries permitted until 2026-07-14 19:14:53.242282001 +0000 UTC m=+246.355531868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/01ac54b9-9511-4e3c-afcd-482292702a0b-service-ca-bundle") pod "router-default-79c9798784-bdl9b" (UID: "01ac54b9-9511-4e3c-afcd-482292702a0b") : configmap references non-existent config key: service-ca.crt Jul 14 19:14:52.242491 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:52.242337 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-metrics-certs podName:01ac54b9-9511-4e3c-afcd-482292702a0b nodeName:}" failed. No retries permitted until 2026-07-14 19:14:53.242318172 +0000 UTC m=+246.355568020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-metrics-certs") pod "router-default-79c9798784-bdl9b" (UID: "01ac54b9-9511-4e3c-afcd-482292702a0b") : secret "router-metrics-certs-default" not found Jul 14 19:14:52.242491 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:52.242359 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c90a7de7-c072-40bd-bebc-ee4996a2e763-cluster-monitoring-operator-tls podName:c90a7de7-c072-40bd-bebc-ee4996a2e763 nodeName:}" failed. No retries permitted until 2026-07-14 19:14:53.242350266 +0000 UTC m=+246.355600115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c90a7de7-c072-40bd-bebc-ee4996a2e763-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-4mvrd" (UID: "c90a7de7-c072-40bd-bebc-ee4996a2e763") : secret "cluster-monitoring-operator-tls" not found Jul 14 19:14:53.250282 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:53.250247 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ac54b9-9511-4e3c-afcd-482292702a0b-service-ca-bundle\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:53.250791 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:53.250298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c90a7de7-c072-40bd-bebc-ee4996a2e763-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-4mvrd\" (UID: \"c90a7de7-c072-40bd-bebc-ee4996a2e763\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" Jul 14 19:14:53.250791 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:53.250331 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-metrics-certs\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:53.250791 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:53.250437 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/01ac54b9-9511-4e3c-afcd-482292702a0b-service-ca-bundle podName:01ac54b9-9511-4e3c-afcd-482292702a0b nodeName:}" failed. No retries permitted until 2026-07-14 19:14:55.250412453 +0000 UTC m=+248.363662320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/01ac54b9-9511-4e3c-afcd-482292702a0b-service-ca-bundle") pod "router-default-79c9798784-bdl9b" (UID: "01ac54b9-9511-4e3c-afcd-482292702a0b") : configmap references non-existent config key: service-ca.crt Jul 14 19:14:53.250791 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:53.250548 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 19:14:53.250791 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:53.250634 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c90a7de7-c072-40bd-bebc-ee4996a2e763-cluster-monitoring-operator-tls podName:c90a7de7-c072-40bd-bebc-ee4996a2e763 nodeName:}" failed. No retries permitted until 2026-07-14 19:14:55.250616575 +0000 UTC m=+248.363866429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c90a7de7-c072-40bd-bebc-ee4996a2e763-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-4mvrd" (UID: "c90a7de7-c072-40bd-bebc-ee4996a2e763") : secret "cluster-monitoring-operator-tls" not found Jul 14 19:14:53.251024 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:53.250795 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 19:14:53.251024 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:53.250845 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-metrics-certs podName:01ac54b9-9511-4e3c-afcd-482292702a0b nodeName:}" failed. No retries permitted until 2026-07-14 19:14:55.250831074 +0000 UTC m=+248.364080935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-metrics-certs") pod "router-default-79c9798784-bdl9b" (UID: "01ac54b9-9511-4e3c-afcd-482292702a0b") : secret "router-metrics-certs-default" not found Jul 14 19:14:55.198314 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:55.198107 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b" event={"ID":"b20459ce-1f34-4dcd-becb-522fee08d273","Type":"ContainerStarted","Data":"e5b9a3fe12d6e8ac3fe913937c7e0947acdce0120e437a0b28c7bd58f926a343"} Jul 14 19:14:55.200012 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:55.199982 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" event={"ID":"e92e640a-57c7-4a12-89cb-701c6f9bf3e6","Type":"ContainerStarted","Data":"41fa49cae61ca06f0d7dcde29b71af5471004e7f6469c1ef5b3af3c5793d9b26"} Jul 14 19:14:55.201822 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:55.201787 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd" event={"ID":"c048bba9-c26d-4398-a1ce-9b4a9be0ab89","Type":"ContainerStarted","Data":"3d39e4b6ce89d4c5035f6a10a0dcc388def1b8f4b16105bc29a47ad87c241527"} Jul 14 19:14:55.215880 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:55.215820 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b" podStartSLOduration=1.257650322 podStartE2EDuration="4.215799973s" totalImagesPullingTime="2.958149651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:14:55.214415731 +0000 UTC m=+248.327665601" watchObservedRunningTime="2026-07-14 19:14:55.215799973 +0000 UTC m=+248.329049844" Jul 14 19:14:55.232318 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:55.232209 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" podStartSLOduration=1.197114033 podStartE2EDuration="4.232186894s" totalImagesPullingTime="3.035072861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:14:55.231658332 +0000 UTC m=+248.344908203" watchObservedRunningTime="2026-07-14 19:14:55.232186894 +0000 UTC m=+248.345436781" Jul 14 19:14:55.252149 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:55.252092 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd" podStartSLOduration=1.294868008 podStartE2EDuration="4.252077426s" totalImagesPullingTime="2.957209418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:14:55.251831851 +0000 UTC m=+248.365081724" watchObservedRunningTime="2026-07-14 19:14:55.252077426 +0000 UTC m=+248.365327309" Jul 14 19:14:55.270546 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:55.270507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ac54b9-9511-4e3c-afcd-482292702a0b-service-ca-bundle\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:55.270711 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:55.270559 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c90a7de7-c072-40bd-bebc-ee4996a2e763-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-4mvrd\" (UID: \"c90a7de7-c072-40bd-bebc-ee4996a2e763\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" Jul 14 19:14:55.270711 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:55.270607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-metrics-certs\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:55.271300 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:55.271279 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 19:14:55.271404 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:55.271342 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c90a7de7-c072-40bd-bebc-ee4996a2e763-cluster-monitoring-operator-tls podName:c90a7de7-c072-40bd-bebc-ee4996a2e763 nodeName:}" failed. No retries permitted until 2026-07-14 19:14:59.271324187 +0000 UTC m=+252.384574038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c90a7de7-c072-40bd-bebc-ee4996a2e763-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-4mvrd" (UID: "c90a7de7-c072-40bd-bebc-ee4996a2e763") : secret "cluster-monitoring-operator-tls" not found Jul 14 19:14:55.271467 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:55.271443 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 19:14:55.271518 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:55.271491 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-metrics-certs podName:01ac54b9-9511-4e3c-afcd-482292702a0b nodeName:}" failed. No retries permitted until 2026-07-14 19:14:59.271476628 +0000 UTC m=+252.384726479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-metrics-certs") pod "router-default-79c9798784-bdl9b" (UID: "01ac54b9-9511-4e3c-afcd-482292702a0b") : secret "router-metrics-certs-default" not found Jul 14 19:14:55.271619 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:55.271606 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/01ac54b9-9511-4e3c-afcd-482292702a0b-service-ca-bundle podName:01ac54b9-9511-4e3c-afcd-482292702a0b nodeName:}" failed. No retries permitted until 2026-07-14 19:14:59.271574296 +0000 UTC m=+252.384824162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/01ac54b9-9511-4e3c-afcd-482292702a0b-service-ca-bundle") pod "router-default-79c9798784-bdl9b" (UID: "01ac54b9-9511-4e3c-afcd-482292702a0b") : configmap references non-existent config key: service-ca.crt Jul 14 19:14:56.447460 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.447413 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m2qzw"] Jul 14 19:14:56.450271 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.450245 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:14:56.454230 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.454206 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9hbmm\"" Jul 14 19:14:56.454640 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.454621 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 19:14:56.455264 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.455249 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 19:14:56.475087 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.475060 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m2qzw"] Jul 14 19:14:56.581039 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.581002 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db924ccb-4ed7-4682-98cf-2e696b0d9405-data-volume\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:14:56.581218 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.581195 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db924ccb-4ed7-4682-98cf-2e696b0d9405-crio-socket\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:14:56.581271 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.581259 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpkjt\" (UniqueName: \"kubernetes.io/projected/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-api-access-lpkjt\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:14:56.581373 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.581355 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db924ccb-4ed7-4682-98cf-2e696b0d9405-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:14:56.581416 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.581392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:14:56.682540 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.682509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db924ccb-4ed7-4682-98cf-2e696b0d9405-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:14:56.682540 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.682541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:14:56.682789 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.682573 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db924ccb-4ed7-4682-98cf-2e696b0d9405-data-volume\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:14:56.682789 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.682639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db924ccb-4ed7-4682-98cf-2e696b0d9405-crio-socket\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:14:56.682789 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:56.682655 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 19:14:56.682789 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:56.682688 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 19:14:56.682789 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:56.682714 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-rbac-proxy-cm podName:db924ccb-4ed7-4682-98cf-2e696b0d9405 nodeName:}" failed. No retries permitted until 2026-07-14 19:14:57.182698919 +0000 UTC m=+250.295948768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m2qzw" (UID: "db924ccb-4ed7-4682-98cf-2e696b0d9405") : configmap "kube-rbac-proxy" not found Jul 14 19:14:56.682789 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.682726 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db924ccb-4ed7-4682-98cf-2e696b0d9405-crio-socket\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:14:56.682789 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.682663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpkjt\" (UniqueName: \"kubernetes.io/projected/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-api-access-lpkjt\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:14:56.682789 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:56.682745 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db924ccb-4ed7-4682-98cf-2e696b0d9405-insights-runtime-extractor-tls podName:db924ccb-4ed7-4682-98cf-2e696b0d9405 nodeName:}" failed. No retries permitted until 2026-07-14 19:14:57.18272908 +0000 UTC m=+250.295978945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/db924ccb-4ed7-4682-98cf-2e696b0d9405-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m2qzw" (UID: "db924ccb-4ed7-4682-98cf-2e696b0d9405") : secret "insights-runtime-extractor-tls" not found Jul 14 19:14:56.683063 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.682961 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db924ccb-4ed7-4682-98cf-2e696b0d9405-data-volume\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:14:56.691509 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:56.691485 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpkjt\" (UniqueName: \"kubernetes.io/projected/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-api-access-lpkjt\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:14:57.187159 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:57.187113 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db924ccb-4ed7-4682-98cf-2e696b0d9405-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:14:57.187159 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:57.187162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:14:57.187368 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:57.187258 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 19:14:57.187368 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:57.187265 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 19:14:57.187368 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:57.187309 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-rbac-proxy-cm podName:db924ccb-4ed7-4682-98cf-2e696b0d9405 nodeName:}" failed. No retries permitted until 2026-07-14 19:14:58.187297392 +0000 UTC m=+251.300547240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m2qzw" (UID: "db924ccb-4ed7-4682-98cf-2e696b0d9405") : configmap "kube-rbac-proxy" not found Jul 14 19:14:57.187368 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:57.187323 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db924ccb-4ed7-4682-98cf-2e696b0d9405-insights-runtime-extractor-tls podName:db924ccb-4ed7-4682-98cf-2e696b0d9405 nodeName:}" failed. No retries permitted until 2026-07-14 19:14:58.187316932 +0000 UTC m=+251.300566781 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/db924ccb-4ed7-4682-98cf-2e696b0d9405-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m2qzw" (UID: "db924ccb-4ed7-4682-98cf-2e696b0d9405") : secret "insights-runtime-extractor-tls" not found Jul 14 19:14:58.195838 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.195801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db924ccb-4ed7-4682-98cf-2e696b0d9405-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:14:58.195838 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.195840 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:14:58.196322 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:58.195984 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 19:14:58.196322 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:58.195996 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 19:14:58.196322 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:58.196065 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db924ccb-4ed7-4682-98cf-2e696b0d9405-insights-runtime-extractor-tls podName:db924ccb-4ed7-4682-98cf-2e696b0d9405 nodeName:}" failed. No retries permitted until 2026-07-14 19:15:00.196045361 +0000 UTC m=+253.309295212 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/db924ccb-4ed7-4682-98cf-2e696b0d9405-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m2qzw" (UID: "db924ccb-4ed7-4682-98cf-2e696b0d9405") : secret "insights-runtime-extractor-tls" not found Jul 14 19:14:58.196322 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:58.196087 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-rbac-proxy-cm podName:db924ccb-4ed7-4682-98cf-2e696b0d9405 nodeName:}" failed. No retries permitted until 2026-07-14 19:15:00.196076627 +0000 UTC m=+253.309326475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m2qzw" (UID: "db924ccb-4ed7-4682-98cf-2e696b0d9405") : configmap "kube-rbac-proxy" not found Jul 14 19:14:58.428784 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.428755 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q6rws_e0a0b990-bc45-4764-bd78-31069184c58a/dns-node-resolver/0.log" Jul 14 19:14:58.698638 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.698601 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-gqnqn"] Jul 14 19:14:58.701631 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.701607 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-gqnqn" Jul 14 19:14:58.705118 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.705087 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 14 19:14:58.705118 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.705083 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 14 19:14:58.705977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.705962 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-gffmw\"" Jul 14 19:14:58.706082 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.706014 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 14 19:14:58.706263 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.706249 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 14 19:14:58.711817 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.711796 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-gqnqn"] Jul 14 19:14:58.800898 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.800867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pztvp\" (UniqueName: \"kubernetes.io/projected/76c4a53b-0b29-4de9-85f1-a13e8a749bdb-kube-api-access-pztvp\") pod \"service-ca-79f68b9f4f-gqnqn\" (UID: \"76c4a53b-0b29-4de9-85f1-a13e8a749bdb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-gqnqn" Jul 14 19:14:58.801058 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.800937 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/76c4a53b-0b29-4de9-85f1-a13e8a749bdb-signing-cabundle\") pod \"service-ca-79f68b9f4f-gqnqn\" (UID: \"76c4a53b-0b29-4de9-85f1-a13e8a749bdb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-gqnqn" Jul 14 19:14:58.801058 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.801018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/76c4a53b-0b29-4de9-85f1-a13e8a749bdb-signing-key\") pod \"service-ca-79f68b9f4f-gqnqn\" (UID: \"76c4a53b-0b29-4de9-85f1-a13e8a749bdb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-gqnqn" Jul 14 19:14:58.902164 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.902128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pztvp\" (UniqueName: \"kubernetes.io/projected/76c4a53b-0b29-4de9-85f1-a13e8a749bdb-kube-api-access-pztvp\") pod \"service-ca-79f68b9f4f-gqnqn\" (UID: \"76c4a53b-0b29-4de9-85f1-a13e8a749bdb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-gqnqn" Jul 14 19:14:58.902338 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.902197 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/76c4a53b-0b29-4de9-85f1-a13e8a749bdb-signing-cabundle\") pod \"service-ca-79f68b9f4f-gqnqn\" (UID: \"76c4a53b-0b29-4de9-85f1-a13e8a749bdb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-gqnqn" Jul 14 19:14:58.902338 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.902282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/76c4a53b-0b29-4de9-85f1-a13e8a749bdb-signing-key\") pod \"service-ca-79f68b9f4f-gqnqn\" (UID: \"76c4a53b-0b29-4de9-85f1-a13e8a749bdb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-gqnqn" Jul 14 19:14:58.902888 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.902868 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/76c4a53b-0b29-4de9-85f1-a13e8a749bdb-signing-cabundle\") pod \"service-ca-79f68b9f4f-gqnqn\" (UID: \"76c4a53b-0b29-4de9-85f1-a13e8a749bdb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-gqnqn" Jul 14 19:14:58.904724 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.904700 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/76c4a53b-0b29-4de9-85f1-a13e8a749bdb-signing-key\") pod \"service-ca-79f68b9f4f-gqnqn\" (UID: \"76c4a53b-0b29-4de9-85f1-a13e8a749bdb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-gqnqn" Jul 14 19:14:58.911244 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:58.911219 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pztvp\" (UniqueName: \"kubernetes.io/projected/76c4a53b-0b29-4de9-85f1-a13e8a749bdb-kube-api-access-pztvp\") pod \"service-ca-79f68b9f4f-gqnqn\" (UID: \"76c4a53b-0b29-4de9-85f1-a13e8a749bdb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-gqnqn" Jul 14 19:14:59.010990 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:59.010919 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-gqnqn" Jul 14 19:14:59.129921 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:59.129890 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-gqnqn"] Jul 14 19:14:59.132556 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:14:59.132533 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c4a53b_0b29_4de9_85f1_a13e8a749bdb.slice/crio-ed392a0032bd567a168645e55a6b621b9e2366bcebb71c49610933dd2ea90987 WatchSource:0}: Error finding container ed392a0032bd567a168645e55a6b621b9e2366bcebb71c49610933dd2ea90987: Status 404 returned error can't find the container with id ed392a0032bd567a168645e55a6b621b9e2366bcebb71c49610933dd2ea90987 Jul 14 19:14:59.205566 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:59.205529 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs\") pod \"network-metrics-daemon-xczln\" (UID: \"8d1706ff-a844-42e9-b6d8-628495d88055\") " pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:14:59.205971 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:59.205698 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 14 19:14:59.205971 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:59.205758 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs podName:8d1706ff-a844-42e9-b6d8-628495d88055 nodeName:}" failed. No retries permitted until 2026-07-14 19:17:01.205743939 +0000 UTC m=+374.318993787 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs") pod "network-metrics-daemon-xczln" (UID: "8d1706ff-a844-42e9-b6d8-628495d88055") : secret "metrics-daemon-secret" not found Jul 14 19:14:59.216020 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:59.215983 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-gqnqn" event={"ID":"76c4a53b-0b29-4de9-85f1-a13e8a749bdb","Type":"ContainerStarted","Data":"3aad85055eebca4ad10972112ce440bcf30b019b998a30da7875b3ce62ff8cb1"} Jul 14 19:14:59.216201 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:59.216043 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-gqnqn" event={"ID":"76c4a53b-0b29-4de9-85f1-a13e8a749bdb","Type":"ContainerStarted","Data":"ed392a0032bd567a168645e55a6b621b9e2366bcebb71c49610933dd2ea90987"} Jul 14 19:14:59.234861 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:59.234811 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-gqnqn" podStartSLOduration=1.234792141 podStartE2EDuration="1.234792141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:14:59.234477966 +0000 UTC m=+252.347727837" watchObservedRunningTime="2026-07-14 19:14:59.234792141 +0000 UTC m=+252.348042053" Jul 14 19:14:59.306365 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:59.306272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ac54b9-9511-4e3c-afcd-482292702a0b-service-ca-bundle\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:59.306365 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:59.306306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c90a7de7-c072-40bd-bebc-ee4996a2e763-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-4mvrd\" (UID: \"c90a7de7-c072-40bd-bebc-ee4996a2e763\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" Jul 14 19:14:59.306567 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:59.306404 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 19:14:59.306567 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:59.306448 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c90a7de7-c072-40bd-bebc-ee4996a2e763-cluster-monitoring-operator-tls podName:c90a7de7-c072-40bd-bebc-ee4996a2e763 nodeName:}" failed. No retries permitted until 2026-07-14 19:15:07.306436378 +0000 UTC m=+260.419686227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c90a7de7-c072-40bd-bebc-ee4996a2e763-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-4mvrd" (UID: "c90a7de7-c072-40bd-bebc-ee4996a2e763") : secret "cluster-monitoring-operator-tls" not found Jul 14 19:14:59.306567 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:59.306453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-metrics-certs\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:14:59.306796 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:59.306598 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/01ac54b9-9511-4e3c-afcd-482292702a0b-service-ca-bundle podName:01ac54b9-9511-4e3c-afcd-482292702a0b nodeName:}" failed. No retries permitted until 2026-07-14 19:15:07.306562562 +0000 UTC m=+260.419812431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/01ac54b9-9511-4e3c-afcd-482292702a0b-service-ca-bundle") pod "router-default-79c9798784-bdl9b" (UID: "01ac54b9-9511-4e3c-afcd-482292702a0b") : configmap references non-existent config key: service-ca.crt Jul 14 19:14:59.306796 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:59.306646 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 19:14:59.306796 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:14:59.306690 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-metrics-certs podName:01ac54b9-9511-4e3c-afcd-482292702a0b nodeName:}" failed. No retries permitted until 2026-07-14 19:15:07.306676904 +0000 UTC m=+260.419926756 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-metrics-certs") pod "router-default-79c9798784-bdl9b" (UID: "01ac54b9-9511-4e3c-afcd-482292702a0b") : secret "router-metrics-certs-default" not found Jul 14 19:14:59.630188 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:14:59.630161 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z4lnk_70375e8e-ae6d-4ce1-8240-012361054757/node-ca/0.log" Jul 14 19:15:00.215414 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:00.215373 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db924ccb-4ed7-4682-98cf-2e696b0d9405-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:15:00.215414 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:00.215417 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:15:00.215925 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:15:00.215498 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 19:15:00.215925 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:15:00.215543 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-rbac-proxy-cm podName:db924ccb-4ed7-4682-98cf-2e696b0d9405 nodeName:}" failed. No retries permitted until 2026-07-14 19:15:04.215531159 +0000 UTC m=+257.328781007 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m2qzw" (UID: "db924ccb-4ed7-4682-98cf-2e696b0d9405") : configmap "kube-rbac-proxy" not found Jul 14 19:15:00.215925 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:15:00.215573 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 19:15:00.215925 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:15:00.215662 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db924ccb-4ed7-4682-98cf-2e696b0d9405-insights-runtime-extractor-tls podName:db924ccb-4ed7-4682-98cf-2e696b0d9405 nodeName:}" failed. No retries permitted until 2026-07-14 19:15:04.215645165 +0000 UTC m=+257.328895012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/db924ccb-4ed7-4682-98cf-2e696b0d9405-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m2qzw" (UID: "db924ccb-4ed7-4682-98cf-2e696b0d9405") : secret "insights-runtime-extractor-tls" not found Jul 14 19:15:00.229067 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:00.229037 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j7zhz_5e789e3e-9025-425e-bdfb-b8caedfa2767/serve-healthcheck-canary/0.log" Jul 14 19:15:04.247745 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:04.247707 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db924ccb-4ed7-4682-98cf-2e696b0d9405-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:15:04.247745 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:04.247748 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:15:04.248174 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:15:04.247826 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 19:15:04.248174 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:15:04.247858 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 19:15:04.248174 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:15:04.247872 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-rbac-proxy-cm podName:db924ccb-4ed7-4682-98cf-2e696b0d9405 nodeName:}" failed. No retries permitted until 2026-07-14 19:15:12.247860577 +0000 UTC m=+265.361110426 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m2qzw" (UID: "db924ccb-4ed7-4682-98cf-2e696b0d9405") : configmap "kube-rbac-proxy" not found Jul 14 19:15:04.248174 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:15:04.247907 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db924ccb-4ed7-4682-98cf-2e696b0d9405-insights-runtime-extractor-tls podName:db924ccb-4ed7-4682-98cf-2e696b0d9405 nodeName:}" failed. No retries permitted until 2026-07-14 19:15:12.247893756 +0000 UTC m=+265.361143604 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/db924ccb-4ed7-4682-98cf-2e696b0d9405-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m2qzw" (UID: "db924ccb-4ed7-4682-98cf-2e696b0d9405") : secret "insights-runtime-extractor-tls" not found Jul 14 19:15:07.374729 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:07.374689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-metrics-certs\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:15:07.375103 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:07.374804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ac54b9-9511-4e3c-afcd-482292702a0b-service-ca-bundle\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:15:07.375103 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:07.374826 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c90a7de7-c072-40bd-bebc-ee4996a2e763-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-4mvrd\" (UID: \"c90a7de7-c072-40bd-bebc-ee4996a2e763\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" Jul 14 19:15:07.375461 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:07.375443 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ac54b9-9511-4e3c-afcd-482292702a0b-service-ca-bundle\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:15:07.377149 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:07.377120 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c90a7de7-c072-40bd-bebc-ee4996a2e763-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-4mvrd\" (UID: \"c90a7de7-c072-40bd-bebc-ee4996a2e763\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" Jul 14 19:15:07.377259 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:07.377173 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ac54b9-9511-4e3c-afcd-482292702a0b-metrics-certs\") pod \"router-default-79c9798784-bdl9b\" (UID: \"01ac54b9-9511-4e3c-afcd-482292702a0b\") " pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:15:07.489160 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:07.489127 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:15:07.502534 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:07.502503 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" Jul 14 19:15:07.640019 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:07.639991 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79c9798784-bdl9b"] Jul 14 19:15:07.642324 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:15:07.642294 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ac54b9_9511_4e3c_afcd_482292702a0b.slice/crio-1a401ae0d3f0d16396b69c6e04f311197bb8199e512d0894643388e2734fcf3a WatchSource:0}: Error finding container 1a401ae0d3f0d16396b69c6e04f311197bb8199e512d0894643388e2734fcf3a: Status 404 returned error can't find the container with id 1a401ae0d3f0d16396b69c6e04f311197bb8199e512d0894643388e2734fcf3a Jul 14 19:15:07.653390 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:07.653365 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd"] Jul 14 19:15:07.656662 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:15:07.656632 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90a7de7_c072_40bd_bebc_ee4996a2e763.slice/crio-e83b48df2f7e37057cfee439475b5c76e45359dcf2972c18464fbcf137c4e269 WatchSource:0}: Error finding container e83b48df2f7e37057cfee439475b5c76e45359dcf2972c18464fbcf137c4e269: Status 404 returned error can't find the container with id e83b48df2f7e37057cfee439475b5c76e45359dcf2972c18464fbcf137c4e269 Jul 14 19:15:08.244663 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:08.244626 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79c9798784-bdl9b" event={"ID":"01ac54b9-9511-4e3c-afcd-482292702a0b","Type":"ContainerStarted","Data":"2591044a7d50dd4ec44f8921fc2d0a1d50b0575aee3290c5bc364302ec138094"} Jul 14 19:15:08.244846 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:08.244681 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79c9798784-bdl9b" event={"ID":"01ac54b9-9511-4e3c-afcd-482292702a0b","Type":"ContainerStarted","Data":"1a401ae0d3f0d16396b69c6e04f311197bb8199e512d0894643388e2734fcf3a"} Jul 14 19:15:08.245599 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:08.245564 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" event={"ID":"c90a7de7-c072-40bd-bebc-ee4996a2e763","Type":"ContainerStarted","Data":"e83b48df2f7e37057cfee439475b5c76e45359dcf2972c18464fbcf137c4e269"} Jul 14 19:15:08.265547 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:08.265500 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-79c9798784-bdl9b" podStartSLOduration=17.265483831 podStartE2EDuration="17.265483831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:15:08.264947149 +0000 UTC m=+261.378197030" watchObservedRunningTime="2026-07-14 19:15:08.265483831 +0000 UTC m=+261.378733679" Jul 14 19:15:08.490169 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:08.490089 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:15:08.492672 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:08.492648 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:15:09.249899 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:09.249866 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:15:09.251308 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:09.251283 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-79c9798784-bdl9b" Jul 14 19:15:10.253924 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:10.253836 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" event={"ID":"c90a7de7-c072-40bd-bebc-ee4996a2e763","Type":"ContainerStarted","Data":"272f01d4f4096688dc56a084376c17c5e499746a2665125f853b19f25d4b85d5"} Jul 14 19:15:10.271019 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:10.270966 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4mvrd" podStartSLOduration=17.030495243 podStartE2EDuration="19.27094484s" totalImagesPullingTime="2.240449597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:15:10.270498154 +0000 UTC m=+263.383748031" watchObservedRunningTime="2026-07-14 19:15:10.27094484 +0000 UTC m=+263.384194713" Jul 14 19:15:12.322323 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:12.322278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db924ccb-4ed7-4682-98cf-2e696b0d9405-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:15:12.322323 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:12.322323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:15:12.322875 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:12.322856 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db924ccb-4ed7-4682-98cf-2e696b0d9405-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:15:12.324577 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:12.324557 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db924ccb-4ed7-4682-98cf-2e696b0d9405-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qzw\" (UID: \"db924ccb-4ed7-4682-98cf-2e696b0d9405\") " pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:15:12.360747 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:12.360709 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m2qzw" Jul 14 19:15:12.484987 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:12.484956 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m2qzw"] Jul 14 19:15:12.487853 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:15:12.487825 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb924ccb_4ed7_4682_98cf_2e696b0d9405.slice/crio-a15689bd6c9a46b3d83729ad2b74060f92a0b71474682ebcdb85fb5e2b219f71 WatchSource:0}: Error finding container a15689bd6c9a46b3d83729ad2b74060f92a0b71474682ebcdb85fb5e2b219f71: Status 404 returned error can't find the container with id a15689bd6c9a46b3d83729ad2b74060f92a0b71474682ebcdb85fb5e2b219f71 Jul 14 19:15:13.266700 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:13.266664 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m2qzw" event={"ID":"db924ccb-4ed7-4682-98cf-2e696b0d9405","Type":"ContainerStarted","Data":"a15689bd6c9a46b3d83729ad2b74060f92a0b71474682ebcdb85fb5e2b219f71"} Jul 14 19:15:17.281068 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:17.280986 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m2qzw" event={"ID":"db924ccb-4ed7-4682-98cf-2e696b0d9405","Type":"ContainerStarted","Data":"2b646183afa2e9750d79c831c100179cf91126a28323ee9dbdc7236d2c63c085"} Jul 14 19:15:17.281068 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:17.281038 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m2qzw" event={"ID":"db924ccb-4ed7-4682-98cf-2e696b0d9405","Type":"ContainerStarted","Data":"66f44d615667a8ad27b10926073bc303e867489d2e0c33cea329625a28b8dbec"} Jul 14 19:15:20.295626 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:20.295591 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m2qzw" event={"ID":"db924ccb-4ed7-4682-98cf-2e696b0d9405","Type":"ContainerStarted","Data":"3ad1b456dbd57b04d285907db5e57f7b8515c0a2e25262dc316fa6d0ec84fa0f"} Jul 14 19:15:20.318020 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:20.317965 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m2qzw" podStartSLOduration=17.512779625 podStartE2EDuration="24.317948365s" totalImagesPullingTime="6.80516874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:14:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:15:20.316075246 +0000 UTC m=+273.429325116" watchObservedRunningTime="2026-07-14 19:15:20.317948365 +0000 UTC m=+273.431198235" Jul 14 19:15:21.746017 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.745978 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b4f8cdf57-hdqch"] Jul 14 19:15:21.748872 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.748842 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.758110 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.758089 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 14 19:15:21.758748 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.758731 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 14 19:15:21.758748 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.758741 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 14 19:15:21.759240 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.759226 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-plsql\"" Jul 14 19:15:21.765665 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.765637 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 14 19:15:21.771066 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.771038 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b4f8cdf57-hdqch"] Jul 14 19:15:21.788481 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.788448 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/44fffc73-9677-466d-a63c-7ba76f31f8f5-registry-certificates\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.788481 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.788484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44fffc73-9677-466d-a63c-7ba76f31f8f5-registry-tls\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.788706 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.788509 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/44fffc73-9677-466d-a63c-7ba76f31f8f5-ca-trust-extracted\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.788706 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.788526 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdj82\" (UniqueName: \"kubernetes.io/projected/44fffc73-9677-466d-a63c-7ba76f31f8f5-kube-api-access-wdj82\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.788706 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.788646 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/44fffc73-9677-466d-a63c-7ba76f31f8f5-installation-pull-secrets\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.788706 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.788677 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/44fffc73-9677-466d-a63c-7ba76f31f8f5-image-registry-private-configuration\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.788706 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.788698 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44fffc73-9677-466d-a63c-7ba76f31f8f5-bound-sa-token\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.788880 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.788741 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/44fffc73-9677-466d-a63c-7ba76f31f8f5-trusted-ca\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.889546 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.889507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/44fffc73-9677-466d-a63c-7ba76f31f8f5-registry-certificates\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.889546 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.889543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44fffc73-9677-466d-a63c-7ba76f31f8f5-registry-tls\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.889819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.889565 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/44fffc73-9677-466d-a63c-7ba76f31f8f5-ca-trust-extracted\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.889819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.889609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdj82\" (UniqueName: \"kubernetes.io/projected/44fffc73-9677-466d-a63c-7ba76f31f8f5-kube-api-access-wdj82\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.889819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.889639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/44fffc73-9677-466d-a63c-7ba76f31f8f5-installation-pull-secrets\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.889819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.889671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/44fffc73-9677-466d-a63c-7ba76f31f8f5-image-registry-private-configuration\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.889819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.889692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44fffc73-9677-466d-a63c-7ba76f31f8f5-bound-sa-token\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.890074 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.889848 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/44fffc73-9677-466d-a63c-7ba76f31f8f5-trusted-ca\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.890132 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.890097 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/44fffc73-9677-466d-a63c-7ba76f31f8f5-ca-trust-extracted\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.890618 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.890575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/44fffc73-9677-466d-a63c-7ba76f31f8f5-registry-certificates\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.891102 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.891072 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/44fffc73-9677-466d-a63c-7ba76f31f8f5-trusted-ca\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.892267 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.892230 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44fffc73-9677-466d-a63c-7ba76f31f8f5-registry-tls\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.892267 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.892235 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/44fffc73-9677-466d-a63c-7ba76f31f8f5-installation-pull-secrets\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.892422 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.892399 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/44fffc73-9677-466d-a63c-7ba76f31f8f5-image-registry-private-configuration\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.898678 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.898651 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44fffc73-9677-466d-a63c-7ba76f31f8f5-bound-sa-token\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:21.898948 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:21.898927 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdj82\" (UniqueName: \"kubernetes.io/projected/44fffc73-9677-466d-a63c-7ba76f31f8f5-kube-api-access-wdj82\") pod \"image-registry-5b4f8cdf57-hdqch\" (UID: \"44fffc73-9677-466d-a63c-7ba76f31f8f5\") " pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:22.057695 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:22.057613 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:22.207449 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:22.207417 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b4f8cdf57-hdqch"] Jul 14 19:15:22.210519 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:15:22.210495 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fffc73_9677_466d_a63c_7ba76f31f8f5.slice/crio-588cbc46a4e49a18cccd52613f744d44a8420194b24545e1f493a904e8e43c00 WatchSource:0}: Error finding container 588cbc46a4e49a18cccd52613f744d44a8420194b24545e1f493a904e8e43c00: Status 404 returned error can't find the container with id 588cbc46a4e49a18cccd52613f744d44a8420194b24545e1f493a904e8e43c00 Jul 14 19:15:22.303350 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:22.303316 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" event={"ID":"44fffc73-9677-466d-a63c-7ba76f31f8f5","Type":"ContainerStarted","Data":"2e3ac6d52983ab041de1fa1b69266303531d7f4d08305cfe27b72a0eb7729089"} Jul 14 19:15:22.303514 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:22.303366 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" event={"ID":"44fffc73-9677-466d-a63c-7ba76f31f8f5","Type":"ContainerStarted","Data":"588cbc46a4e49a18cccd52613f744d44a8420194b24545e1f493a904e8e43c00"} Jul 14 19:15:22.303514 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:22.303398 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:22.323193 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:22.323145 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" podStartSLOduration=1.323128845 podStartE2EDuration="1.323128845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:15:22.322662319 +0000 UTC m=+275.435912187" watchObservedRunningTime="2026-07-14 19:15:22.323128845 +0000 UTC m=+275.436378714" Jul 14 19:15:25.251842 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.251810 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6df9dbdbf5-h97k8"] Jul 14 19:15:25.255535 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.255517 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.258357 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.258335 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 14 19:15:25.259373 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.259348 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 14 19:15:25.259493 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.259400 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 14 19:15:25.259493 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.259421 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 14 19:15:25.259493 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.259433 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 14 19:15:25.259493 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.259441 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 14 19:15:25.259493 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.259472 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-s9s5l\"" Jul 14 19:15:25.259773 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.259402 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 14 19:15:25.264937 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.264914 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6df9dbdbf5-h97k8"] Jul 14 19:15:25.315755 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.315717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1e637f7-692d-4334-9ad0-79a5d250479e-console-oauth-config\") pod \"console-6df9dbdbf5-h97k8\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.315755 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.315762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1e637f7-692d-4334-9ad0-79a5d250479e-oauth-serving-cert\") pod \"console-6df9dbdbf5-h97k8\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.315971 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.315802 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bfrf\" (UniqueName: \"kubernetes.io/projected/e1e637f7-692d-4334-9ad0-79a5d250479e-kube-api-access-7bfrf\") pod \"console-6df9dbdbf5-h97k8\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.315971 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.315837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e637f7-692d-4334-9ad0-79a5d250479e-service-ca\") pod \"console-6df9dbdbf5-h97k8\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.315971 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.315872 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1e637f7-692d-4334-9ad0-79a5d250479e-console-serving-cert\") pod \"console-6df9dbdbf5-h97k8\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.315971 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.315902 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1e637f7-692d-4334-9ad0-79a5d250479e-console-config\") pod \"console-6df9dbdbf5-h97k8\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.417263 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.417226 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1e637f7-692d-4334-9ad0-79a5d250479e-console-oauth-config\") pod \"console-6df9dbdbf5-h97k8\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.417263 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.417268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1e637f7-692d-4334-9ad0-79a5d250479e-oauth-serving-cert\") pod \"console-6df9dbdbf5-h97k8\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.417531 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.417336 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bfrf\" (UniqueName: \"kubernetes.io/projected/e1e637f7-692d-4334-9ad0-79a5d250479e-kube-api-access-7bfrf\") pod \"console-6df9dbdbf5-h97k8\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.417531 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.417401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e637f7-692d-4334-9ad0-79a5d250479e-service-ca\") pod \"console-6df9dbdbf5-h97k8\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.417531 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.417430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1e637f7-692d-4334-9ad0-79a5d250479e-console-serving-cert\") pod \"console-6df9dbdbf5-h97k8\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.417531 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.417466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1e637f7-692d-4334-9ad0-79a5d250479e-console-config\") pod \"console-6df9dbdbf5-h97k8\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.418100 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.418071 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1e637f7-692d-4334-9ad0-79a5d250479e-oauth-serving-cert\") pod \"console-6df9dbdbf5-h97k8\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.418204 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.418145 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e637f7-692d-4334-9ad0-79a5d250479e-service-ca\") pod \"console-6df9dbdbf5-h97k8\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.418264 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.418246 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1e637f7-692d-4334-9ad0-79a5d250479e-console-config\") pod \"console-6df9dbdbf5-h97k8\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.419920 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.419891 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1e637f7-692d-4334-9ad0-79a5d250479e-console-serving-cert\") pod \"console-6df9dbdbf5-h97k8\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.419920 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.419906 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1e637f7-692d-4334-9ad0-79a5d250479e-console-oauth-config\") pod \"console-6df9dbdbf5-h97k8\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.428897 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.428873 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bfrf\" (UniqueName: \"kubernetes.io/projected/e1e637f7-692d-4334-9ad0-79a5d250479e-kube-api-access-7bfrf\") pod \"console-6df9dbdbf5-h97k8\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.566218 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.566127 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:25.701156 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:25.701109 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6df9dbdbf5-h97k8"] Jul 14 19:15:25.703746 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:15:25.703723 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e637f7_692d_4334_9ad0_79a5d250479e.slice/crio-7a81c992829d3e9c75ace63ccacba7267d7dd6acb0821b5c1226a3246d57e77d WatchSource:0}: Error finding container 7a81c992829d3e9c75ace63ccacba7267d7dd6acb0821b5c1226a3246d57e77d: Status 404 returned error can't find the container with id 7a81c992829d3e9c75ace63ccacba7267d7dd6acb0821b5c1226a3246d57e77d Jul 14 19:15:26.315730 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:26.315696 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6df9dbdbf5-h97k8" event={"ID":"e1e637f7-692d-4334-9ad0-79a5d250479e","Type":"ContainerStarted","Data":"7a81c992829d3e9c75ace63ccacba7267d7dd6acb0821b5c1226a3246d57e77d"} Jul 14 19:15:29.328007 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:29.327969 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6df9dbdbf5-h97k8" event={"ID":"e1e637f7-692d-4334-9ad0-79a5d250479e","Type":"ContainerStarted","Data":"55b8d864358bb14791d420bc63ffbe60a259dd99c5e7f0aa499caef4726a1ffe"} Jul 14 19:15:29.347408 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:29.347361 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6df9dbdbf5-h97k8" podStartSLOduration=1.077701871 podStartE2EDuration="4.347345613s" totalImagesPullingTime="3.269643742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:15:29.346388285 +0000 UTC m=+282.459638156" watchObservedRunningTime="2026-07-14 19:15:29.347345613 +0000 UTC m=+282.460595482" Jul 14 19:15:30.979005 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:15:30.978960 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-29wvt" podUID="f3ea0e40-5a86-4c4b-980b-2007a1290795" Jul 14 19:15:31.334677 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:31.334646 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-29wvt" Jul 14 19:15:34.192899 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.192855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:15:34.195228 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.195204 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3ea0e40-5a86-4c4b-980b-2007a1290795-metrics-tls\") pod \"dns-default-29wvt\" (UID: \"f3ea0e40-5a86-4c4b-980b-2007a1290795\") " pod="openshift-dns/dns-default-29wvt" Jul 14 19:15:34.338886 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.338853 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h5x4l\"" Jul 14 19:15:34.346122 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.346099 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-29wvt" Jul 14 19:15:34.703902 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.703843 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-29wvt"] Jul 14 19:15:34.706621 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:15:34.706560 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ea0e40_5a86_4c4b_980b_2007a1290795.slice/crio-9d4918318b37f3fb1d1ceadbacdf11a555f3982e51c674a7262e5a3621e0fc60 WatchSource:0}: Error finding container 9d4918318b37f3fb1d1ceadbacdf11a555f3982e51c674a7262e5a3621e0fc60: Status 404 returned error can't find the container with id 9d4918318b37f3fb1d1ceadbacdf11a555f3982e51c674a7262e5a3621e0fc60 Jul 14 19:15:34.861718 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.861633 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp"] Jul 14 19:15:34.895895 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.895862 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" Jul 14 19:15:34.898816 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.898650 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 19:15:34.898816 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.898650 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 14 19:15:34.898816 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.898654 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 14 19:15:34.899054 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.898939 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-db8fh\"" Jul 14 19:15:34.914602 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.914546 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp"] Jul 14 19:15:34.915453 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.915431 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-hrqw7"] Jul 14 19:15:34.940009 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.939982 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:34.943191 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.943164 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 14 19:15:34.943337 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.943225 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 14 19:15:34.943405 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.943355 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-w262x\"" Jul 14 19:15:34.943459 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.943444 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 14 19:15:34.945474 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.945448 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zz95x"] Jul 14 19:15:34.971154 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.971125 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-hrqw7"] Jul 14 19:15:34.971282 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.971275 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:34.973883 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.973858 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 19:15:34.974044 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.973935 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 19:15:34.974119 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.973980 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vnvln\"" Jul 14 19:15:34.974188 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.974123 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 19:15:34.999428 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.999392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db281d62-b537-4268-b9f5-7946d58f1daf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-zpfsp\" (UID: \"db281d62-b537-4268-b9f5-7946d58f1daf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" Jul 14 19:15:34.999578 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.999432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13abf531-dedb-495b-9d91-96c55586b544-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:34.999578 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.999468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db281d62-b537-4268-b9f5-7946d58f1daf-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-zpfsp\" (UID: \"db281d62-b537-4268-b9f5-7946d58f1daf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" Jul 14 19:15:34.999578 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.999518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db281d62-b537-4268-b9f5-7946d58f1daf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-zpfsp\" (UID: \"db281d62-b537-4268-b9f5-7946d58f1daf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" Jul 14 19:15:34.999578 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.999544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67gnh\" (UniqueName: \"kubernetes.io/projected/db281d62-b537-4268-b9f5-7946d58f1daf-kube-api-access-67gnh\") pod \"openshift-state-metrics-84bd68b896-zpfsp\" (UID: \"db281d62-b537-4268-b9f5-7946d58f1daf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" Jul 14 19:15:34.999578 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.999574 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/13abf531-dedb-495b-9d91-96c55586b544-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:34.999855 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.999619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/13abf531-dedb-495b-9d91-96c55586b544-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:34.999855 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.999655 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljhs5\" (UniqueName: \"kubernetes.io/projected/13abf531-dedb-495b-9d91-96c55586b544-kube-api-access-ljhs5\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:34.999855 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.999692 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13abf531-dedb-495b-9d91-96c55586b544-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:34.999855 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:34.999745 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/13abf531-dedb-495b-9d91-96c55586b544-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:35.100711 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.100680 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db281d62-b537-4268-b9f5-7946d58f1daf-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-zpfsp\" (UID: \"db281d62-b537-4268-b9f5-7946d58f1daf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" Jul 14 19:15:35.100891 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.100728 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db281d62-b537-4268-b9f5-7946d58f1daf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-zpfsp\" (UID: \"db281d62-b537-4268-b9f5-7946d58f1daf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" Jul 14 19:15:35.100891 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.100755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67gnh\" (UniqueName: \"kubernetes.io/projected/db281d62-b537-4268-b9f5-7946d58f1daf-kube-api-access-67gnh\") pod \"openshift-state-metrics-84bd68b896-zpfsp\" (UID: \"db281d62-b537-4268-b9f5-7946d58f1daf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" Jul 14 19:15:35.100891 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.100775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-textfile\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.100891 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.100795 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/13abf531-dedb-495b-9d91-96c55586b544-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:35.100891 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.100820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/13abf531-dedb-495b-9d91-96c55586b544-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:35.100891 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.100847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljhs5\" (UniqueName: \"kubernetes.io/projected/13abf531-dedb-495b-9d91-96c55586b544-kube-api-access-ljhs5\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:35.100891 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.100862 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fd24d679-7f30-4b07-9d49-24f9f81bfb58-root\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.100891 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.100880 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-wtmp\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.101355 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.100935 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-accelerators-collector-config\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.101355 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.100961 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfdlw\" (UniqueName: \"kubernetes.io/projected/fd24d679-7f30-4b07-9d49-24f9f81bfb58-kube-api-access-kfdlw\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.101355 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:15:35.101046 2576 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 14 19:15:35.101355 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:15:35.101124 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13abf531-dedb-495b-9d91-96c55586b544-kube-state-metrics-tls podName:13abf531-dedb-495b-9d91-96c55586b544 nodeName:}" failed. No retries permitted until 2026-07-14 19:15:35.601102753 +0000 UTC m=+288.714352601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/13abf531-dedb-495b-9d91-96c55586b544-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-hrqw7" (UID: "13abf531-dedb-495b-9d91-96c55586b544") : secret "kube-state-metrics-tls" not found Jul 14 19:15:35.101355 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.101234 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/13abf531-dedb-495b-9d91-96c55586b544-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:35.101355 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.101334 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.101680 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.101386 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13abf531-dedb-495b-9d91-96c55586b544-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:35.101680 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.101421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/13abf531-dedb-495b-9d91-96c55586b544-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:35.101680 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.101450 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-tls\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.101680 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.101471 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db281d62-b537-4268-b9f5-7946d58f1daf-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-zpfsp\" (UID: \"db281d62-b537-4268-b9f5-7946d58f1daf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" Jul 14 19:15:35.101680 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.101502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd24d679-7f30-4b07-9d49-24f9f81bfb58-sys\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.101680 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.101533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db281d62-b537-4268-b9f5-7946d58f1daf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-zpfsp\" (UID: \"db281d62-b537-4268-b9f5-7946d58f1daf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" Jul 14 19:15:35.101680 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.101572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13abf531-dedb-495b-9d91-96c55586b544-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:35.101680 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.101626 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd24d679-7f30-4b07-9d49-24f9f81bfb58-metrics-client-ca\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.102072 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.102054 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/13abf531-dedb-495b-9d91-96c55586b544-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:35.102286 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.102266 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13abf531-dedb-495b-9d91-96c55586b544-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:35.103734 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.103713 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db281d62-b537-4268-b9f5-7946d58f1daf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-zpfsp\" (UID: \"db281d62-b537-4268-b9f5-7946d58f1daf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" Jul 14 19:15:35.103979 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.103961 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db281d62-b537-4268-b9f5-7946d58f1daf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-zpfsp\" (UID: \"db281d62-b537-4268-b9f5-7946d58f1daf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" Jul 14 19:15:35.104044 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.104008 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13abf531-dedb-495b-9d91-96c55586b544-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:35.110157 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.110129 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67gnh\" (UniqueName: \"kubernetes.io/projected/db281d62-b537-4268-b9f5-7946d58f1daf-kube-api-access-67gnh\") pod \"openshift-state-metrics-84bd68b896-zpfsp\" (UID: \"db281d62-b537-4268-b9f5-7946d58f1daf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" Jul 14 19:15:35.110256 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.110183 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljhs5\" (UniqueName: \"kubernetes.io/projected/13abf531-dedb-495b-9d91-96c55586b544-kube-api-access-ljhs5\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:35.202461 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.202382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-textfile\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.202461 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.202434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fd24d679-7f30-4b07-9d49-24f9f81bfb58-root\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.202944 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.202488 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fd24d679-7f30-4b07-9d49-24f9f81bfb58-root\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.202944 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.202518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-wtmp\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.202944 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.202541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-accelerators-collector-config\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.202944 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.202560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfdlw\" (UniqueName: \"kubernetes.io/projected/fd24d679-7f30-4b07-9d49-24f9f81bfb58-kube-api-access-kfdlw\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.202944 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.202601 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.202944 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.202641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-tls\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.202944 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.202720 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-wtmp\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.202944 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:15:35.202747 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 14 19:15:35.202944 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.202757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd24d679-7f30-4b07-9d49-24f9f81bfb58-sys\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.202944 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:15:35.202800 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-tls podName:fd24d679-7f30-4b07-9d49-24f9f81bfb58 nodeName:}" failed. No retries permitted until 2026-07-14 19:15:35.702781355 +0000 UTC m=+288.816031207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-tls") pod "node-exporter-zz95x" (UID: "fd24d679-7f30-4b07-9d49-24f9f81bfb58") : secret "node-exporter-tls" not found Jul 14 19:15:35.202944 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.202809 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd24d679-7f30-4b07-9d49-24f9f81bfb58-sys\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.202944 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.202815 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd24d679-7f30-4b07-9d49-24f9f81bfb58-metrics-client-ca\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.202944 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.202818 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-textfile\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.203370 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.203179 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-accelerators-collector-config\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.203370 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.203288 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd24d679-7f30-4b07-9d49-24f9f81bfb58-metrics-client-ca\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.205009 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.204989 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.205900 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.205884 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" Jul 14 19:15:35.212461 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.212441 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfdlw\" (UniqueName: \"kubernetes.io/projected/fd24d679-7f30-4b07-9d49-24f9f81bfb58-kube-api-access-kfdlw\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.349435 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.349397 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-29wvt" event={"ID":"f3ea0e40-5a86-4c4b-980b-2007a1290795","Type":"ContainerStarted","Data":"9d4918318b37f3fb1d1ceadbacdf11a555f3982e51c674a7262e5a3621e0fc60"} Jul 14 19:15:35.362400 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.362324 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp"] Jul 14 19:15:35.366763 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:15:35.366699 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb281d62_b537_4268_b9f5_7946d58f1daf.slice/crio-340a625ab30993b9c9b11dff2d5d4d65757a65869ed338edafe54d79f33d8ba9 WatchSource:0}: Error finding container 340a625ab30993b9c9b11dff2d5d4d65757a65869ed338edafe54d79f33d8ba9: Status 404 returned error can't find the container with id 340a625ab30993b9c9b11dff2d5d4d65757a65869ed338edafe54d79f33d8ba9 Jul 14 19:15:35.566811 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.566778 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:35.566979 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.566845 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:15:35.568317 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.568284 2576 patch_prober.go:28] interesting pod/console-6df9dbdbf5-h97k8 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.17:8443/health\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body= Jul 14 19:15:35.568430 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.568338 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6df9dbdbf5-h97k8" podUID="e1e637f7-692d-4334-9ad0-79a5d250479e" containerName="console" probeResult="failure" output="Get \"https://10.132.0.17:8443/health\": dial tcp 10.132.0.17:8443: connect: connection refused" Jul 14 19:15:35.607642 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.607612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/13abf531-dedb-495b-9d91-96c55586b544-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:35.610541 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.610509 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/13abf531-dedb-495b-9d91-96c55586b544-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-hrqw7\" (UID: \"13abf531-dedb-495b-9d91-96c55586b544\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:35.708795 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.708699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-tls\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:35.709081 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:15:35.708872 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 14 19:15:35.709081 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:15:35.708949 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-tls podName:fd24d679-7f30-4b07-9d49-24f9f81bfb58 nodeName:}" failed. No retries permitted until 2026-07-14 19:15:36.708928873 +0000 UTC m=+289.822178742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-tls") pod "node-exporter-zz95x" (UID: "fd24d679-7f30-4b07-9d49-24f9f81bfb58") : secret "node-exporter-tls" not found Jul 14 19:15:35.849920 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.849887 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" Jul 14 19:15:35.913624 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.913416 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 19:15:35.917975 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.917953 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:35.923696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.923202 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 14 19:15:35.923696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.923462 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 14 19:15:35.923696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.923505 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-94brn\"" Jul 14 19:15:35.923696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.923684 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 14 19:15:35.923981 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.923859 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 14 19:15:35.925195 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.924249 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 14 19:15:35.925195 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.924447 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 14 19:15:35.925195 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.924671 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 14 19:15:35.925195 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.924858 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 14 19:15:35.925195 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.925044 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 14 19:15:35.934080 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:35.934055 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 19:15:36.011139 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.011055 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5b672f03-23ea-450b-83a1-9b6d8f8de338-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.011139 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.011104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b672f03-23ea-450b-83a1-9b6d8f8de338-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.011351 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.011204 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.011351 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.011240 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q2r8\" (UniqueName: \"kubernetes.io/projected/5b672f03-23ea-450b-83a1-9b6d8f8de338-kube-api-access-7q2r8\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.011351 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.011267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-config-volume\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.011502 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.011366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.011502 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.011416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b672f03-23ea-450b-83a1-9b6d8f8de338-config-out\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.011502 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.011460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b672f03-23ea-450b-83a1-9b6d8f8de338-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.011502 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.011484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.011723 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.011527 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-web-config\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.011723 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.011556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.011723 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.011602 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.011723 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.011647 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b672f03-23ea-450b-83a1-9b6d8f8de338-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.019772 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.019744 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-hrqw7"] Jul 14 19:15:36.022006 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:15:36.021983 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13abf531_dedb_495b_9d91_96c55586b544.slice/crio-caef76c1a4c9ab9ef3bb2b5e50d90b0931205321c2d43a20c533b49b3bb70f23 WatchSource:0}: Error finding container caef76c1a4c9ab9ef3bb2b5e50d90b0931205321c2d43a20c533b49b3bb70f23: Status 404 returned error can't find the container with id caef76c1a4c9ab9ef3bb2b5e50d90b0931205321c2d43a20c533b49b3bb70f23 Jul 14 19:15:36.112636 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.112553 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.112810 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.112646 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b672f03-23ea-450b-83a1-9b6d8f8de338-config-out\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.112810 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.112689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b672f03-23ea-450b-83a1-9b6d8f8de338-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.112810 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.112714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.112810 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.112755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-web-config\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.112810 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.112784 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.112810 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.112809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.113143 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.112853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b672f03-23ea-450b-83a1-9b6d8f8de338-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.113143 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.112885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5b672f03-23ea-450b-83a1-9b6d8f8de338-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.113143 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.112911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b672f03-23ea-450b-83a1-9b6d8f8de338-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.113143 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.112966 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.113143 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.112990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q2r8\" (UniqueName: \"kubernetes.io/projected/5b672f03-23ea-450b-83a1-9b6d8f8de338-kube-api-access-7q2r8\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.113143 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.113020 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-config-volume\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.114071 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.114043 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5b672f03-23ea-450b-83a1-9b6d8f8de338-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.115551 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.115522 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b672f03-23ea-450b-83a1-9b6d8f8de338-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.115671 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:15:36.115647 2576 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 14 19:15:36.115736 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:15:36.115713 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-main-tls podName:5b672f03-23ea-450b-83a1-9b6d8f8de338 nodeName:}" failed. No retries permitted until 2026-07-14 19:15:36.615697414 +0000 UTC m=+289.728947270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "5b672f03-23ea-450b-83a1-9b6d8f8de338") : secret "alertmanager-main-tls" not found Jul 14 19:15:36.115943 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.115920 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b672f03-23ea-450b-83a1-9b6d8f8de338-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.117366 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.116522 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-config-volume\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.117366 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.116947 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.117366 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.117325 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.117905 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.117881 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b672f03-23ea-450b-83a1-9b6d8f8de338-config-out\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.118837 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.118811 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.118939 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.118818 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b672f03-23ea-450b-83a1-9b6d8f8de338-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.120429 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.120344 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-web-config\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.120964 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.120939 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.124625 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.124603 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q2r8\" (UniqueName: \"kubernetes.io/projected/5b672f03-23ea-450b-83a1-9b6d8f8de338-kube-api-access-7q2r8\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.355354 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.355301 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" event={"ID":"13abf531-dedb-495b-9d91-96c55586b544","Type":"ContainerStarted","Data":"caef76c1a4c9ab9ef3bb2b5e50d90b0931205321c2d43a20c533b49b3bb70f23"} Jul 14 19:15:36.357595 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.357550 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" event={"ID":"db281d62-b537-4268-b9f5-7946d58f1daf","Type":"ContainerStarted","Data":"cf170f4633a8864503d44d894ed391b8964f9a4a74df52ace4e6d0594e38c7af"} Jul 14 19:15:36.357704 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.357621 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" event={"ID":"db281d62-b537-4268-b9f5-7946d58f1daf","Type":"ContainerStarted","Data":"8d40cf8d41a3e67001f7ebef642187609c961f9f8c56891b992c7e6bbc590385"} Jul 14 19:15:36.357704 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.357653 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" event={"ID":"db281d62-b537-4268-b9f5-7946d58f1daf","Type":"ContainerStarted","Data":"340a625ab30993b9c9b11dff2d5d4d65757a65869ed338edafe54d79f33d8ba9"} Jul 14 19:15:36.619166 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.619066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.622339 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.622298 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.720320 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.720282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-tls\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:36.723068 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.723046 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd24d679-7f30-4b07-9d49-24f9f81bfb58-node-exporter-tls\") pod \"node-exporter-zz95x\" (UID: \"fd24d679-7f30-4b07-9d49-24f9f81bfb58\") " pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:36.782394 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.782356 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zz95x" Jul 14 19:15:36.835576 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:36.835537 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:15:36.940240 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:15:36.940208 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd24d679_7f30_4b07_9d49_24f9f81bfb58.slice/crio-988f9bc5774e6715573c2f1421a694b859a0acdd841b550a66204a36f81ca01b WatchSource:0}: Error finding container 988f9bc5774e6715573c2f1421a694b859a0acdd841b550a66204a36f81ca01b: Status 404 returned error can't find the container with id 988f9bc5774e6715573c2f1421a694b859a0acdd841b550a66204a36f81ca01b Jul 14 19:15:37.099839 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:37.099817 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 19:15:37.102239 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:15:37.102171 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b672f03_23ea_450b_83a1_9b6d8f8de338.slice/crio-a356f7a2fb8cab4ab148b22b2e745b5d3abd2ab2e42cbca6d958a29af9eca6a1 WatchSource:0}: Error finding container a356f7a2fb8cab4ab148b22b2e745b5d3abd2ab2e42cbca6d958a29af9eca6a1: Status 404 returned error can't find the container with id a356f7a2fb8cab4ab148b22b2e745b5d3abd2ab2e42cbca6d958a29af9eca6a1 Jul 14 19:15:37.364270 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:37.364227 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-29wvt" event={"ID":"f3ea0e40-5a86-4c4b-980b-2007a1290795","Type":"ContainerStarted","Data":"63a46dd6afa6011a72d3c13a08b8c956f3cf2cbe498e54278400bc7791a4642e"} Jul 14 19:15:37.364784 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:37.364290 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-29wvt" event={"ID":"f3ea0e40-5a86-4c4b-980b-2007a1290795","Type":"ContainerStarted","Data":"eac0a81a6586b7bf164b6ca7f8cfeeb8dcd5996bf1155ca21c83049b99111b95"} Jul 14 19:15:37.364784 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:37.364333 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-29wvt" Jul 14 19:15:37.365794 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:37.365767 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b672f03-23ea-450b-83a1-9b6d8f8de338","Type":"ContainerStarted","Data":"a356f7a2fb8cab4ab148b22b2e745b5d3abd2ab2e42cbca6d958a29af9eca6a1"} Jul 14 19:15:37.368057 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:37.368031 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" event={"ID":"db281d62-b537-4268-b9f5-7946d58f1daf","Type":"ContainerStarted","Data":"1e483aaf3526df38e3a1e1b1db11662317cd5aafef99d66fa8fcefea2b9195a6"} Jul 14 19:15:37.369296 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:37.369276 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zz95x" event={"ID":"fd24d679-7f30-4b07-9d49-24f9f81bfb58","Type":"ContainerStarted","Data":"988f9bc5774e6715573c2f1421a694b859a0acdd841b550a66204a36f81ca01b"} Jul 14 19:15:37.380760 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:37.380700 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-29wvt" podStartSLOduration=251.148849244 podStartE2EDuration="4m13.380682697s" totalImagesPullingTime="2.231833453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:15:37.379934944 +0000 UTC m=+290.493184815" watchObservedRunningTime="2026-07-14 19:15:37.380682697 +0000 UTC m=+290.493932568" Jul 14 19:15:37.401910 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:37.401849 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zpfsp" podStartSLOduration=1.946974021 podStartE2EDuration="3.401829979s" totalImagesPullingTime="1.454855958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:15:37.401181288 +0000 UTC m=+290.514431170" watchObservedRunningTime="2026-07-14 19:15:37.401829979 +0000 UTC m=+290.515079849" Jul 14 19:15:38.376972 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:38.376925 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" event={"ID":"13abf531-dedb-495b-9d91-96c55586b544","Type":"ContainerStarted","Data":"89cf0646a7e4b3f54212e9525a5783ae33b659a39038b92756fdc6d699943ff2"} Jul 14 19:15:38.377411 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:38.377036 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" event={"ID":"13abf531-dedb-495b-9d91-96c55586b544","Type":"ContainerStarted","Data":"504b313d55c6b5fa1bd0343fb996e139717e1d0ed3a386a940077ca852c09861"} Jul 14 19:15:38.377411 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:38.377070 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" event={"ID":"13abf531-dedb-495b-9d91-96c55586b544","Type":"ContainerStarted","Data":"4163b0ef76825d8bb2f53326b460a28d3a765685872c540b36785d06760f319a"} Jul 14 19:15:38.399494 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:38.399441 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-hrqw7" podStartSLOduration=2.91575854 podStartE2EDuration="4.39942494s" totalImagesPullingTime="1.4836664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:15:38.397229062 +0000 UTC m=+291.510478934" watchObservedRunningTime="2026-07-14 19:15:38.39942494 +0000 UTC m=+291.512674809" Jul 14 19:15:39.382249 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:39.382211 2576 generic.go:356] "Generic (PLEG): container finished" podID="fd24d679-7f30-4b07-9d49-24f9f81bfb58" containerID="bcffcaf334a19411db76ac2d1af511aab5ed6b779b4fb5f95bd2324465046eec" exitCode=0 Jul 14 19:15:39.382687 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:39.382294 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zz95x" event={"ID":"fd24d679-7f30-4b07-9d49-24f9f81bfb58","Type":"ContainerDied","Data":"bcffcaf334a19411db76ac2d1af511aab5ed6b779b4fb5f95bd2324465046eec"} Jul 14 19:15:39.383746 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:39.383726 2576 generic.go:356] "Generic (PLEG): container finished" podID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerID="337b09272f81bf1bb04a79328d664f54655801cad3b6d09bf7eb4085ba88ab8a" exitCode=0 Jul 14 19:15:39.383854 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:39.383808 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b672f03-23ea-450b-83a1-9b6d8f8de338","Type":"ContainerDied","Data":"337b09272f81bf1bb04a79328d664f54655801cad3b6d09bf7eb4085ba88ab8a"} Jul 14 19:15:39.607499 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:39.607468 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6df9dbdbf5-h97k8"] Jul 14 19:15:40.390003 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:40.389961 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zz95x" event={"ID":"fd24d679-7f30-4b07-9d49-24f9f81bfb58","Type":"ContainerStarted","Data":"3e4d7c033037f48e48c99d25115ed9d98fedf91e19a194361b43dc336003af36"} Jul 14 19:15:40.390366 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:40.390026 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zz95x" event={"ID":"fd24d679-7f30-4b07-9d49-24f9f81bfb58","Type":"ContainerStarted","Data":"e67848476cd6d4f4068307f2b2758d26208fc27c595a9415b1f5b6115fbeb9d2"} Jul 14 19:15:40.411295 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:40.411247 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zz95x" podStartSLOduration=4.813437635 podStartE2EDuration="6.41122893s" totalImagesPullingTime="1.531648739s" totalInitContainerRuntime="66.142556ms" isStatefulPod=true podCreationTimestamp="2026-07-14 19:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:15:40.409855354 +0000 UTC m=+293.523105226" watchObservedRunningTime="2026-07-14 19:15:40.41122893 +0000 UTC m=+293.524478865" Jul 14 19:15:41.118224 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.118204 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 19:15:41.121784 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.121767 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.124667 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.124643 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 14 19:15:41.124789 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.124664 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 14 19:15:41.124789 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.124669 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 14 19:15:41.124789 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.124642 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 14 19:15:41.125013 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.124998 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 14 19:15:41.125536 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.125518 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 14 19:15:41.126444 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.126412 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 14 19:15:41.127462 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.127073 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 14 19:15:41.132771 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.127714 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 14 19:15:41.132771 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.128002 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dhjdz\"" Jul 14 19:15:41.132771 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.129357 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 14 19:15:41.132771 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.129644 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 14 19:15:41.132771 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.130435 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 14 19:15:41.132771 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.130465 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 14 19:15:41.132771 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.130991 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fgvakbes7hip8\"" Jul 14 19:15:41.132771 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.131115 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 14 19:15:41.135508 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.135486 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 14 19:15:41.139762 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.139741 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 19:15:41.267246 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-config-out\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.267360 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267253 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.267360 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267275 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.267360 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267290 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.267360 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267345 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.267530 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267406 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.267530 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267440 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-config\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.267530 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267463 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.267530 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267490 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.267739 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.267739 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.267739 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.267739 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267666 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7hjj\" (UniqueName: \"kubernetes.io/projected/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-kube-api-access-c7hjj\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.267739 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267694 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.267739 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267734 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.268012 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.268012 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.268012 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267834 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-web-config\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.268012 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267860 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.268012 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.267894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.369316 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.369280 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.369316 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.369322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.369570 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.369349 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.369570 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.369372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-web-config\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.369570 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.369390 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.369570 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.369515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.369797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.369624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-config-out\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.369797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.369654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.369797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.369689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.369797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.369716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.369797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.369748 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.369797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.369794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.370057 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.369824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-config\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.370057 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.369846 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.370057 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.369872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.370057 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.369922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.370057 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.369947 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.370057 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.369973 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.370057 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.370000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7hjj\" (UniqueName: \"kubernetes.io/projected/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-kube-api-access-c7hjj\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.370057 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.370030 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.370453 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.370123 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.370453 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.370431 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.370552 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.370521 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.370875 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.370849 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.371083 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.371061 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.372476 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.372453 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.372768 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.372641 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.372977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.372956 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.373427 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.373167 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.373427 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.373274 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.373427 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.373376 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-config-out\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.373812 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.373792 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.373897 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.373807 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.374450 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.374428 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-web-config\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.374505 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.374433 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.374984 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.374955 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.375350 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.375325 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.375419 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.375388 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.375801 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.375786 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-config\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.379287 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.379272 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7hjj\" (UniqueName: \"kubernetes.io/projected/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-kube-api-access-c7hjj\") pod \"prometheus-k8s-0\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.396694 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.396671 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b672f03-23ea-450b-83a1-9b6d8f8de338","Type":"ContainerStarted","Data":"4c88039476cf4dfa61f9860f82e8eebd32b8e77dbae97fa909c669d704458480"} Jul 14 19:15:41.397002 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.396730 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b672f03-23ea-450b-83a1-9b6d8f8de338","Type":"ContainerStarted","Data":"44c71abd667fb0a23324d4eeb06efc776fef03391f291a59a406c9c249a33a72"} Jul 14 19:15:41.397002 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.396755 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b672f03-23ea-450b-83a1-9b6d8f8de338","Type":"ContainerStarted","Data":"c2186f0ab928f5cc5fad93fb4332644df2350878a90a56ba54d56eb23c981eab"} Jul 14 19:15:41.397002 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.396782 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b672f03-23ea-450b-83a1-9b6d8f8de338","Type":"ContainerStarted","Data":"d0263cc23b3526cefce08631ac2c133c3ac911550965a06ea0af63abf1cbe7c5"} Jul 14 19:15:41.397002 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.396806 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b672f03-23ea-450b-83a1-9b6d8f8de338","Type":"ContainerStarted","Data":"863e7a765e4e2bffca749e2d6164389f72eba487396dd0a69ac0475000ad17d6"} Jul 14 19:15:41.437452 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.437420 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:41.574965 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:41.574882 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 19:15:41.577565 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:15:41.577537 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fb16f4_1dad_47ad_bf02_1269d56f90ac.slice/crio-9abc27ebf27838da108672e171b60a7b20c39446a84e60841883e4f0ece7055c WatchSource:0}: Error finding container 9abc27ebf27838da108672e171b60a7b20c39446a84e60841883e4f0ece7055c: Status 404 returned error can't find the container with id 9abc27ebf27838da108672e171b60a7b20c39446a84e60841883e4f0ece7055c Jul 14 19:15:42.061640 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:42.061609 2576 patch_prober.go:28] interesting pod/image-registry-5b4f8cdf57-hdqch container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 14 19:15:42.061773 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:42.061655 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" podUID="44fffc73-9677-466d-a63c-7ba76f31f8f5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 19:15:42.403148 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:42.403115 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b672f03-23ea-450b-83a1-9b6d8f8de338","Type":"ContainerStarted","Data":"cb950f5e67fe05a6567940531f8ecd4e6367fdcdaf725e34d065ef00fae25993"} Jul 14 19:15:42.404609 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:42.404567 2576 generic.go:356] "Generic (PLEG): container finished" podID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerID="f3075527a8b1f7caff5b3b991bab9f88527b2f9ac93bcf71cd1ab434f8e4ad2b" exitCode=0 Jul 14 19:15:42.404751 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:42.404621 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fb16f4-1dad-47ad-bf02-1269d56f90ac","Type":"ContainerDied","Data":"f3075527a8b1f7caff5b3b991bab9f88527b2f9ac93bcf71cd1ab434f8e4ad2b"} Jul 14 19:15:42.404751 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:42.404682 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fb16f4-1dad-47ad-bf02-1269d56f90ac","Type":"ContainerStarted","Data":"9abc27ebf27838da108672e171b60a7b20c39446a84e60841883e4f0ece7055c"} Jul 14 19:15:42.431060 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:42.431006 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.717051982 podStartE2EDuration="7.430990903s" totalImagesPullingTime="3.695225186s" totalInitContainerRuntime="18.713735ms" isStatefulPod=false podCreationTimestamp="2026-07-14 19:15:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:15:42.42916101 +0000 UTC m=+295.542410880" watchObservedRunningTime="2026-07-14 19:15:42.430990903 +0000 UTC m=+295.544240773" Jul 14 19:15:43.310184 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:43.310151 2576 patch_prober.go:28] interesting pod/image-registry-5b4f8cdf57-hdqch container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 14 19:15:43.310388 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:43.310237 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" podUID="44fffc73-9677-466d-a63c-7ba76f31f8f5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 19:15:45.509728 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:15:45.509688 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xczln" podUID="8d1706ff-a844-42e9-b6d8-628495d88055" Jul 14 19:15:46.424879 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:46.424848 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fb16f4-1dad-47ad-bf02-1269d56f90ac","Type":"ContainerStarted","Data":"a4a3cb7b89940f91d7ecf1e427c053c004ca70d0fa182e8d2d24942b1e22930d"} Jul 14 19:15:46.425037 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:46.424935 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fb16f4-1dad-47ad-bf02-1269d56f90ac","Type":"ContainerStarted","Data":"07abb4b61c660e64f2bae6b1841f2e4c4fe8dfabbdb1a3969b35307d1ef41f2a"} Jul 14 19:15:47.371817 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:47.371789 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:15:47.372947 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:47.372919 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:15:47.374674 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:47.374654 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 19:15:47.379738 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:47.379713 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-29wvt" Jul 14 19:15:48.442415 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:48.442330 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fb16f4-1dad-47ad-bf02-1269d56f90ac","Type":"ContainerStarted","Data":"b15e6116f94841672d982f6a15385a31838ac02e9efbd04449d132f5612bebe4"} Jul 14 19:15:48.442813 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:48.442413 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fb16f4-1dad-47ad-bf02-1269d56f90ac","Type":"ContainerStarted","Data":"1316e1ed01f270fd6e9480765b1c99a87b7fa1c3819df2f1624a4b4734f3b64b"} Jul 14 19:15:48.442813 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:48.442456 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fb16f4-1dad-47ad-bf02-1269d56f90ac","Type":"ContainerStarted","Data":"ca6d57a56e0b77f4fbc12051f9bc1d156cb91266290cc1e3eac137d3bdeaef0a"} Jul 14 19:15:48.442813 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:48.442491 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fb16f4-1dad-47ad-bf02-1269d56f90ac","Type":"ContainerStarted","Data":"28753fdb9c82d1d176bb5f8c3bea514e119773c4d8e77e11c364f0f515b1be4b"} Jul 14 19:15:48.473117 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:48.473057 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.082161395 podStartE2EDuration="7.473034659s" totalImagesPullingTime="5.369739477s" totalInitContainerRuntime="21.133787ms" isStatefulPod=false podCreationTimestamp="2026-07-14 19:15:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:15:48.470503844 +0000 UTC m=+301.583753713" watchObservedRunningTime="2026-07-14 19:15:48.473034659 +0000 UTC m=+301.586284530" Jul 14 19:15:51.438103 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:51.438065 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:15:52.062326 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:52.062289 2576 patch_prober.go:28] interesting pod/image-registry-5b4f8cdf57-hdqch container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 14 19:15:52.062515 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:52.062351 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" podUID="44fffc73-9677-466d-a63c-7ba76f31f8f5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 19:15:53.310658 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:53.310630 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b4f8cdf57-hdqch" Jul 14 19:15:59.509045 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:15:59.508999 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:16:01.496888 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:01.496856 2576 generic.go:356] "Generic (PLEG): container finished" podID="e92e640a-57c7-4a12-89cb-701c6f9bf3e6" containerID="41fa49cae61ca06f0d7dcde29b71af5471004e7f6469c1ef5b3af3c5793d9b26" exitCode=0 Jul 14 19:16:01.497266 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:01.496916 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" event={"ID":"e92e640a-57c7-4a12-89cb-701c6f9bf3e6","Type":"ContainerDied","Data":"41fa49cae61ca06f0d7dcde29b71af5471004e7f6469c1ef5b3af3c5793d9b26"} Jul 14 19:16:01.497344 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:01.497330 2576 scope.go:117] "RemoveContainer" containerID="41fa49cae61ca06f0d7dcde29b71af5471004e7f6469c1ef5b3af3c5793d9b26" Jul 14 19:16:02.190128 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:02.190085 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79c9798784-bdl9b_01ac54b9-9511-4e3c-afcd-482292702a0b/router/0.log" Jul 14 19:16:02.202037 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:02.202004 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j7zhz_5e789e3e-9025-425e-bdfb-b8caedfa2767/serve-healthcheck-canary/0.log" Jul 14 19:16:02.503276 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:02.503197 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-pxckb" event={"ID":"e92e640a-57c7-4a12-89cb-701c6f9bf3e6","Type":"ContainerStarted","Data":"d5e8e71e128266de602e3e54ce23671d21e86ae36034951dc54066c4fd2402de"} Jul 14 19:16:04.632001 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.631954 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6df9dbdbf5-h97k8" podUID="e1e637f7-692d-4334-9ad0-79a5d250479e" containerName="console" containerID="cri-o://55b8d864358bb14791d420bc63ffbe60a259dd99c5e7f0aa499caef4726a1ffe" gracePeriod=15 Jul 14 19:16:04.872368 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.872338 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:16:04.891305 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.891219 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1e637f7-692d-4334-9ad0-79a5d250479e-oauth-serving-cert\") pod \"e1e637f7-692d-4334-9ad0-79a5d250479e\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " Jul 14 19:16:04.891305 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.891299 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bfrf\" (UniqueName: \"kubernetes.io/projected/e1e637f7-692d-4334-9ad0-79a5d250479e-kube-api-access-7bfrf\") pod \"e1e637f7-692d-4334-9ad0-79a5d250479e\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " Jul 14 19:16:04.891516 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.891339 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1e637f7-692d-4334-9ad0-79a5d250479e-console-serving-cert\") pod \"e1e637f7-692d-4334-9ad0-79a5d250479e\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " Jul 14 19:16:04.891516 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.891392 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1e637f7-692d-4334-9ad0-79a5d250479e-console-config\") pod \"e1e637f7-692d-4334-9ad0-79a5d250479e\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " Jul 14 19:16:04.891516 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.891433 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1e637f7-692d-4334-9ad0-79a5d250479e-console-oauth-config\") pod \"e1e637f7-692d-4334-9ad0-79a5d250479e\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " Jul 14 19:16:04.891516 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.891468 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e637f7-692d-4334-9ad0-79a5d250479e-service-ca\") pod \"e1e637f7-692d-4334-9ad0-79a5d250479e\" (UID: \"e1e637f7-692d-4334-9ad0-79a5d250479e\") " Jul 14 19:16:04.891764 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.891742 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e637f7-692d-4334-9ad0-79a5d250479e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e1e637f7-692d-4334-9ad0-79a5d250479e" (UID: "e1e637f7-692d-4334-9ad0-79a5d250479e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:16:04.891945 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.891834 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1e637f7-692d-4334-9ad0-79a5d250479e-oauth-serving-cert\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:04.892116 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.892037 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e637f7-692d-4334-9ad0-79a5d250479e-service-ca" (OuterVolumeSpecName: "service-ca") pod "e1e637f7-692d-4334-9ad0-79a5d250479e" (UID: "e1e637f7-692d-4334-9ad0-79a5d250479e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:16:04.892266 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.892241 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e637f7-692d-4334-9ad0-79a5d250479e-console-config" (OuterVolumeSpecName: "console-config") pod "e1e637f7-692d-4334-9ad0-79a5d250479e" (UID: "e1e637f7-692d-4334-9ad0-79a5d250479e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:16:04.894896 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.894859 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e637f7-692d-4334-9ad0-79a5d250479e-kube-api-access-7bfrf" (OuterVolumeSpecName: "kube-api-access-7bfrf") pod "e1e637f7-692d-4334-9ad0-79a5d250479e" (UID: "e1e637f7-692d-4334-9ad0-79a5d250479e"). InnerVolumeSpecName "kube-api-access-7bfrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:16:04.895018 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.894964 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e637f7-692d-4334-9ad0-79a5d250479e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e1e637f7-692d-4334-9ad0-79a5d250479e" (UID: "e1e637f7-692d-4334-9ad0-79a5d250479e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:04.895074 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.895049 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e637f7-692d-4334-9ad0-79a5d250479e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e1e637f7-692d-4334-9ad0-79a5d250479e" (UID: "e1e637f7-692d-4334-9ad0-79a5d250479e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:04.992800 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.992760 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bfrf\" (UniqueName: \"kubernetes.io/projected/e1e637f7-692d-4334-9ad0-79a5d250479e-kube-api-access-7bfrf\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:04.992800 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.992793 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1e637f7-692d-4334-9ad0-79a5d250479e-console-serving-cert\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:04.992800 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.992804 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1e637f7-692d-4334-9ad0-79a5d250479e-console-config\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:04.992800 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.992812 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1e637f7-692d-4334-9ad0-79a5d250479e-console-oauth-config\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:04.993076 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:04.992821 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e637f7-692d-4334-9ad0-79a5d250479e-service-ca\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:05.515708 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:05.515674 2576 generic.go:356] "Generic (PLEG): container finished" podID="e1e637f7-692d-4334-9ad0-79a5d250479e" containerID="55b8d864358bb14791d420bc63ffbe60a259dd99c5e7f0aa499caef4726a1ffe" exitCode=0 Jul 14 19:16:05.515863 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:05.515715 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6df9dbdbf5-h97k8" Jul 14 19:16:05.515863 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:05.515752 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6df9dbdbf5-h97k8" event={"ID":"e1e637f7-692d-4334-9ad0-79a5d250479e","Type":"ContainerDied","Data":"55b8d864358bb14791d420bc63ffbe60a259dd99c5e7f0aa499caef4726a1ffe"} Jul 14 19:16:05.515863 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:05.515796 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6df9dbdbf5-h97k8" event={"ID":"e1e637f7-692d-4334-9ad0-79a5d250479e","Type":"ContainerDied","Data":"7a81c992829d3e9c75ace63ccacba7267d7dd6acb0821b5c1226a3246d57e77d"} Jul 14 19:16:05.515863 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:05.515815 2576 scope.go:117] "RemoveContainer" containerID="55b8d864358bb14791d420bc63ffbe60a259dd99c5e7f0aa499caef4726a1ffe" Jul 14 19:16:05.525573 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:05.525550 2576 scope.go:117] "RemoveContainer" containerID="55b8d864358bb14791d420bc63ffbe60a259dd99c5e7f0aa499caef4726a1ffe" Jul 14 19:16:05.525995 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:16:05.525965 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b8d864358bb14791d420bc63ffbe60a259dd99c5e7f0aa499caef4726a1ffe\": container with ID starting with 55b8d864358bb14791d420bc63ffbe60a259dd99c5e7f0aa499caef4726a1ffe not found: ID does not exist" containerID="55b8d864358bb14791d420bc63ffbe60a259dd99c5e7f0aa499caef4726a1ffe" Jul 14 19:16:05.526087 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:05.526005 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b8d864358bb14791d420bc63ffbe60a259dd99c5e7f0aa499caef4726a1ffe"} err="failed to get container status \"55b8d864358bb14791d420bc63ffbe60a259dd99c5e7f0aa499caef4726a1ffe\": rpc error: code = NotFound desc = could not find container \"55b8d864358bb14791d420bc63ffbe60a259dd99c5e7f0aa499caef4726a1ffe\": container with ID starting with 55b8d864358bb14791d420bc63ffbe60a259dd99c5e7f0aa499caef4726a1ffe not found: ID does not exist" Jul 14 19:16:05.538839 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:05.538807 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6df9dbdbf5-h97k8"] Jul 14 19:16:05.543243 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:05.543217 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6df9dbdbf5-h97k8"] Jul 14 19:16:07.514365 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:07.514335 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1e637f7-692d-4334-9ad0-79a5d250479e" path="/var/lib/kubelet/pods/e1e637f7-692d-4334-9ad0-79a5d250479e/volumes" Jul 14 19:16:16.565278 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:16.565231 2576 generic.go:356] "Generic (PLEG): container finished" podID="c048bba9-c26d-4398-a1ce-9b4a9be0ab89" containerID="3d39e4b6ce89d4c5035f6a10a0dcc388def1b8f4b16105bc29a47ad87c241527" exitCode=0 Jul 14 19:16:16.565764 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:16.565307 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd" event={"ID":"c048bba9-c26d-4398-a1ce-9b4a9be0ab89","Type":"ContainerDied","Data":"3d39e4b6ce89d4c5035f6a10a0dcc388def1b8f4b16105bc29a47ad87c241527"} Jul 14 19:16:16.565764 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:16.565710 2576 scope.go:117] "RemoveContainer" containerID="3d39e4b6ce89d4c5035f6a10a0dcc388def1b8f4b16105bc29a47ad87c241527" Jul 14 19:16:16.566762 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:16.566736 2576 generic.go:356] "Generic (PLEG): container finished" podID="b20459ce-1f34-4dcd-becb-522fee08d273" containerID="e5b9a3fe12d6e8ac3fe913937c7e0947acdce0120e437a0b28c7bd58f926a343" exitCode=0 Jul 14 19:16:16.566873 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:16.566805 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b" event={"ID":"b20459ce-1f34-4dcd-becb-522fee08d273","Type":"ContainerDied","Data":"e5b9a3fe12d6e8ac3fe913937c7e0947acdce0120e437a0b28c7bd58f926a343"} Jul 14 19:16:16.567073 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:16.567058 2576 scope.go:117] "RemoveContainer" containerID="e5b9a3fe12d6e8ac3fe913937c7e0947acdce0120e437a0b28c7bd58f926a343" Jul 14 19:16:17.572723 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:17.572688 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rq5xd" event={"ID":"c048bba9-c26d-4398-a1ce-9b4a9be0ab89","Type":"ContainerStarted","Data":"2ca2f43e994be5f3a3626c41799e2d4b5e0a0299745d728caff89396c3a9459f"} Jul 14 19:16:17.574563 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:17.574537 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dnr7b" event={"ID":"b20459ce-1f34-4dcd-becb-522fee08d273","Type":"ContainerStarted","Data":"46d48ac979bc56354d57b3f9f85f0214da005930fc0d335f1d2a13870b195f8a"} Jul 14 19:16:41.437898 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:41.437802 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:16:41.453559 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:41.453530 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:16:41.680235 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:41.680211 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:16:55.217789 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:55.217752 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 19:16:55.218231 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:55.218059 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="alertmanager" containerID="cri-o://863e7a765e4e2bffca749e2d6164389f72eba487396dd0a69ac0475000ad17d6" gracePeriod=120 Jul 14 19:16:55.218231 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:55.218147 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="kube-rbac-proxy-web" containerID="cri-o://c2186f0ab928f5cc5fad93fb4332644df2350878a90a56ba54d56eb23c981eab" gracePeriod=120 Jul 14 19:16:55.218231 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:55.218141 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="kube-rbac-proxy-metric" containerID="cri-o://4c88039476cf4dfa61f9860f82e8eebd32b8e77dbae97fa909c669d704458480" gracePeriod=120 Jul 14 19:16:55.218231 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:55.218193 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="prom-label-proxy" containerID="cri-o://cb950f5e67fe05a6567940531f8ecd4e6367fdcdaf725e34d065ef00fae25993" gracePeriod=120 Jul 14 19:16:55.218444 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:55.218238 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="config-reloader" containerID="cri-o://d0263cc23b3526cefce08631ac2c133c3ac911550965a06ea0af63abf1cbe7c5" gracePeriod=120 Jul 14 19:16:55.218444 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:55.218274 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="kube-rbac-proxy" containerID="cri-o://44c71abd667fb0a23324d4eeb06efc776fef03391f291a59a406c9c249a33a72" gracePeriod=120 Jul 14 19:16:55.724493 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:55.724454 2576 generic.go:356] "Generic (PLEG): container finished" podID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerID="cb950f5e67fe05a6567940531f8ecd4e6367fdcdaf725e34d065ef00fae25993" exitCode=0 Jul 14 19:16:55.724493 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:55.724478 2576 generic.go:356] "Generic (PLEG): container finished" podID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerID="44c71abd667fb0a23324d4eeb06efc776fef03391f291a59a406c9c249a33a72" exitCode=0 Jul 14 19:16:55.724493 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:55.724484 2576 generic.go:356] "Generic (PLEG): container finished" podID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerID="d0263cc23b3526cefce08631ac2c133c3ac911550965a06ea0af63abf1cbe7c5" exitCode=0 Jul 14 19:16:55.724493 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:55.724491 2576 generic.go:356] "Generic (PLEG): container finished" podID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerID="863e7a765e4e2bffca749e2d6164389f72eba487396dd0a69ac0475000ad17d6" exitCode=0 Jul 14 19:16:55.724839 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:55.724537 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b672f03-23ea-450b-83a1-9b6d8f8de338","Type":"ContainerDied","Data":"cb950f5e67fe05a6567940531f8ecd4e6367fdcdaf725e34d065ef00fae25993"} Jul 14 19:16:55.724839 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:55.724660 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b672f03-23ea-450b-83a1-9b6d8f8de338","Type":"ContainerDied","Data":"44c71abd667fb0a23324d4eeb06efc776fef03391f291a59a406c9c249a33a72"} Jul 14 19:16:55.724839 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:55.724703 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b672f03-23ea-450b-83a1-9b6d8f8de338","Type":"ContainerDied","Data":"d0263cc23b3526cefce08631ac2c133c3ac911550965a06ea0af63abf1cbe7c5"} Jul 14 19:16:55.724839 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:55.724743 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b672f03-23ea-450b-83a1-9b6d8f8de338","Type":"ContainerDied","Data":"863e7a765e4e2bffca749e2d6164389f72eba487396dd0a69ac0475000ad17d6"} Jul 14 19:16:56.470001 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.469972 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:16:56.557683 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.557599 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-web-config\") pod \"5b672f03-23ea-450b-83a1-9b6d8f8de338\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " Jul 14 19:16:56.557683 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.557667 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b672f03-23ea-450b-83a1-9b6d8f8de338-alertmanager-trusted-ca-bundle\") pod \"5b672f03-23ea-450b-83a1-9b6d8f8de338\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " Jul 14 19:16:56.557914 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.557700 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-kube-rbac-proxy-web\") pod \"5b672f03-23ea-450b-83a1-9b6d8f8de338\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " Jul 14 19:16:56.557914 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.557737 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b672f03-23ea-450b-83a1-9b6d8f8de338-metrics-client-ca\") pod \"5b672f03-23ea-450b-83a1-9b6d8f8de338\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " Jul 14 19:16:56.557914 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.557771 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-config-volume\") pod \"5b672f03-23ea-450b-83a1-9b6d8f8de338\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " Jul 14 19:16:56.557914 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.557809 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-main-tls\") pod \"5b672f03-23ea-450b-83a1-9b6d8f8de338\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " Jul 14 19:16:56.557914 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.557844 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5b672f03-23ea-450b-83a1-9b6d8f8de338-alertmanager-main-db\") pod \"5b672f03-23ea-450b-83a1-9b6d8f8de338\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " Jul 14 19:16:56.557914 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.557868 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b672f03-23ea-450b-83a1-9b6d8f8de338-config-out\") pod \"5b672f03-23ea-450b-83a1-9b6d8f8de338\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " Jul 14 19:16:56.557914 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.557898 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q2r8\" (UniqueName: \"kubernetes.io/projected/5b672f03-23ea-450b-83a1-9b6d8f8de338-kube-api-access-7q2r8\") pod \"5b672f03-23ea-450b-83a1-9b6d8f8de338\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " Jul 14 19:16:56.558234 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.557937 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b672f03-23ea-450b-83a1-9b6d8f8de338-tls-assets\") pod \"5b672f03-23ea-450b-83a1-9b6d8f8de338\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " Jul 14 19:16:56.558234 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.557970 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5b672f03-23ea-450b-83a1-9b6d8f8de338\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " Jul 14 19:16:56.558234 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.557988 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-cluster-tls-config\") pod \"5b672f03-23ea-450b-83a1-9b6d8f8de338\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " Jul 14 19:16:56.558234 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.558014 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-kube-rbac-proxy\") pod \"5b672f03-23ea-450b-83a1-9b6d8f8de338\" (UID: \"5b672f03-23ea-450b-83a1-9b6d8f8de338\") " Jul 14 19:16:56.558234 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.558183 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b672f03-23ea-450b-83a1-9b6d8f8de338-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "5b672f03-23ea-450b-83a1-9b6d8f8de338" (UID: "5b672f03-23ea-450b-83a1-9b6d8f8de338"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:16:56.558469 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.558295 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b672f03-23ea-450b-83a1-9b6d8f8de338-metrics-client-ca\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:56.558557 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.558522 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b672f03-23ea-450b-83a1-9b6d8f8de338-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "5b672f03-23ea-450b-83a1-9b6d8f8de338" (UID: "5b672f03-23ea-450b-83a1-9b6d8f8de338"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:16:56.558637 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.558540 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b672f03-23ea-450b-83a1-9b6d8f8de338-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "5b672f03-23ea-450b-83a1-9b6d8f8de338" (UID: "5b672f03-23ea-450b-83a1-9b6d8f8de338"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:16:56.560721 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.560686 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "5b672f03-23ea-450b-83a1-9b6d8f8de338" (UID: "5b672f03-23ea-450b-83a1-9b6d8f8de338"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:56.561017 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.560989 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-config-volume" (OuterVolumeSpecName: "config-volume") pod "5b672f03-23ea-450b-83a1-9b6d8f8de338" (UID: "5b672f03-23ea-450b-83a1-9b6d8f8de338"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:56.561210 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.561179 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b672f03-23ea-450b-83a1-9b6d8f8de338-kube-api-access-7q2r8" (OuterVolumeSpecName: "kube-api-access-7q2r8") pod "5b672f03-23ea-450b-83a1-9b6d8f8de338" (UID: "5b672f03-23ea-450b-83a1-9b6d8f8de338"). InnerVolumeSpecName "kube-api-access-7q2r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:16:56.561294 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.561250 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b672f03-23ea-450b-83a1-9b6d8f8de338-config-out" (OuterVolumeSpecName: "config-out") pod "5b672f03-23ea-450b-83a1-9b6d8f8de338" (UID: "5b672f03-23ea-450b-83a1-9b6d8f8de338"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:16:56.561337 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.561299 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "5b672f03-23ea-450b-83a1-9b6d8f8de338" (UID: "5b672f03-23ea-450b-83a1-9b6d8f8de338"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:56.561490 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.561465 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "5b672f03-23ea-450b-83a1-9b6d8f8de338" (UID: "5b672f03-23ea-450b-83a1-9b6d8f8de338"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:56.561903 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.561876 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "5b672f03-23ea-450b-83a1-9b6d8f8de338" (UID: "5b672f03-23ea-450b-83a1-9b6d8f8de338"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:56.562739 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.562715 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b672f03-23ea-450b-83a1-9b6d8f8de338-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5b672f03-23ea-450b-83a1-9b6d8f8de338" (UID: "5b672f03-23ea-450b-83a1-9b6d8f8de338"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:16:56.566350 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.566328 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "5b672f03-23ea-450b-83a1-9b6d8f8de338" (UID: "5b672f03-23ea-450b-83a1-9b6d8f8de338"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:56.572739 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.572712 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-web-config" (OuterVolumeSpecName: "web-config") pod "5b672f03-23ea-450b-83a1-9b6d8f8de338" (UID: "5b672f03-23ea-450b-83a1-9b6d8f8de338"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:56.659277 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.659241 2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-config-volume\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:56.659277 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.659270 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-main-tls\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:56.659277 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.659282 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5b672f03-23ea-450b-83a1-9b6d8f8de338-alertmanager-main-db\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:56.659504 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.659290 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b672f03-23ea-450b-83a1-9b6d8f8de338-config-out\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:56.659504 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.659301 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q2r8\" (UniqueName: \"kubernetes.io/projected/5b672f03-23ea-450b-83a1-9b6d8f8de338-kube-api-access-7q2r8\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:56.659504 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.659309 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b672f03-23ea-450b-83a1-9b6d8f8de338-tls-assets\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:56.659504 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.659317 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:56.659504 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.659326 2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-cluster-tls-config\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:56.659504 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.659334 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:56.659504 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.659343 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-web-config\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:56.659504 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.659352 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b672f03-23ea-450b-83a1-9b6d8f8de338-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:56.659504 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.659360 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b672f03-23ea-450b-83a1-9b6d8f8de338-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:56.730941 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.730907 2576 generic.go:356] "Generic (PLEG): container finished" podID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerID="4c88039476cf4dfa61f9860f82e8eebd32b8e77dbae97fa909c669d704458480" exitCode=0 Jul 14 19:16:56.730941 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.730939 2576 generic.go:356] "Generic (PLEG): container finished" podID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerID="c2186f0ab928f5cc5fad93fb4332644df2350878a90a56ba54d56eb23c981eab" exitCode=0 Jul 14 19:16:56.731185 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.730992 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b672f03-23ea-450b-83a1-9b6d8f8de338","Type":"ContainerDied","Data":"4c88039476cf4dfa61f9860f82e8eebd32b8e77dbae97fa909c669d704458480"} Jul 14 19:16:56.731185 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.731062 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b672f03-23ea-450b-83a1-9b6d8f8de338","Type":"ContainerDied","Data":"c2186f0ab928f5cc5fad93fb4332644df2350878a90a56ba54d56eb23c981eab"} Jul 14 19:16:56.731185 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.731002 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 19:16:56.731185 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.731089 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b672f03-23ea-450b-83a1-9b6d8f8de338","Type":"ContainerDied","Data":"a356f7a2fb8cab4ab148b22b2e745b5d3abd2ab2e42cbca6d958a29af9eca6a1"} Jul 14 19:16:56.731185 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.731125 2576 scope.go:117] "RemoveContainer" containerID="cb950f5e67fe05a6567940531f8ecd4e6367fdcdaf725e34d065ef00fae25993" Jul 14 19:16:56.739802 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.739781 2576 scope.go:117] "RemoveContainer" containerID="4c88039476cf4dfa61f9860f82e8eebd32b8e77dbae97fa909c669d704458480" Jul 14 19:16:56.746489 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.746471 2576 scope.go:117] "RemoveContainer" containerID="44c71abd667fb0a23324d4eeb06efc776fef03391f291a59a406c9c249a33a72" Jul 14 19:16:56.753160 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.753138 2576 scope.go:117] "RemoveContainer" containerID="c2186f0ab928f5cc5fad93fb4332644df2350878a90a56ba54d56eb23c981eab" Jul 14 19:16:56.756369 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.756349 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 19:16:56.759533 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.759512 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 19:16:56.761280 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.761265 2576 scope.go:117] "RemoveContainer" containerID="d0263cc23b3526cefce08631ac2c133c3ac911550965a06ea0af63abf1cbe7c5" Jul 14 19:16:56.768439 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.768398 2576 scope.go:117] "RemoveContainer" containerID="863e7a765e4e2bffca749e2d6164389f72eba487396dd0a69ac0475000ad17d6" Jul 14 19:16:56.775612 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.775577 2576 scope.go:117] "RemoveContainer" containerID="337b09272f81bf1bb04a79328d664f54655801cad3b6d09bf7eb4085ba88ab8a" Jul 14 19:16:56.783269 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.783250 2576 scope.go:117] "RemoveContainer" containerID="cb950f5e67fe05a6567940531f8ecd4e6367fdcdaf725e34d065ef00fae25993" Jul 14 19:16:56.783528 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:16:56.783507 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb950f5e67fe05a6567940531f8ecd4e6367fdcdaf725e34d065ef00fae25993\": container with ID starting with cb950f5e67fe05a6567940531f8ecd4e6367fdcdaf725e34d065ef00fae25993 not found: ID does not exist" containerID="cb950f5e67fe05a6567940531f8ecd4e6367fdcdaf725e34d065ef00fae25993" Jul 14 19:16:56.783570 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.783537 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb950f5e67fe05a6567940531f8ecd4e6367fdcdaf725e34d065ef00fae25993"} err="failed to get container status \"cb950f5e67fe05a6567940531f8ecd4e6367fdcdaf725e34d065ef00fae25993\": rpc error: code = NotFound desc = could not find container \"cb950f5e67fe05a6567940531f8ecd4e6367fdcdaf725e34d065ef00fae25993\": container with ID starting with cb950f5e67fe05a6567940531f8ecd4e6367fdcdaf725e34d065ef00fae25993 not found: ID does not exist" Jul 14 19:16:56.783570 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.783555 2576 scope.go:117] "RemoveContainer" containerID="4c88039476cf4dfa61f9860f82e8eebd32b8e77dbae97fa909c669d704458480" Jul 14 19:16:56.783842 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:16:56.783825 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c88039476cf4dfa61f9860f82e8eebd32b8e77dbae97fa909c669d704458480\": container with ID starting with 4c88039476cf4dfa61f9860f82e8eebd32b8e77dbae97fa909c669d704458480 not found: ID does not exist" containerID="4c88039476cf4dfa61f9860f82e8eebd32b8e77dbae97fa909c669d704458480" Jul 14 19:16:56.783884 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.783848 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c88039476cf4dfa61f9860f82e8eebd32b8e77dbae97fa909c669d704458480"} err="failed to get container status \"4c88039476cf4dfa61f9860f82e8eebd32b8e77dbae97fa909c669d704458480\": rpc error: code = NotFound desc = could not find container \"4c88039476cf4dfa61f9860f82e8eebd32b8e77dbae97fa909c669d704458480\": container with ID starting with 4c88039476cf4dfa61f9860f82e8eebd32b8e77dbae97fa909c669d704458480 not found: ID does not exist" Jul 14 19:16:56.783884 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.783864 2576 scope.go:117] "RemoveContainer" containerID="44c71abd667fb0a23324d4eeb06efc776fef03391f291a59a406c9c249a33a72" Jul 14 19:16:56.784074 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:16:56.784058 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c71abd667fb0a23324d4eeb06efc776fef03391f291a59a406c9c249a33a72\": container with ID starting with 44c71abd667fb0a23324d4eeb06efc776fef03391f291a59a406c9c249a33a72 not found: ID does not exist" containerID="44c71abd667fb0a23324d4eeb06efc776fef03391f291a59a406c9c249a33a72" Jul 14 19:16:56.784111 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.784078 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c71abd667fb0a23324d4eeb06efc776fef03391f291a59a406c9c249a33a72"} err="failed to get container status \"44c71abd667fb0a23324d4eeb06efc776fef03391f291a59a406c9c249a33a72\": rpc error: code = NotFound desc = could not find container \"44c71abd667fb0a23324d4eeb06efc776fef03391f291a59a406c9c249a33a72\": container with ID starting with 44c71abd667fb0a23324d4eeb06efc776fef03391f291a59a406c9c249a33a72 not found: ID does not exist" Jul 14 19:16:56.784111 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.784092 2576 scope.go:117] "RemoveContainer" containerID="c2186f0ab928f5cc5fad93fb4332644df2350878a90a56ba54d56eb23c981eab" Jul 14 19:16:56.784537 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:16:56.784501 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2186f0ab928f5cc5fad93fb4332644df2350878a90a56ba54d56eb23c981eab\": container with ID starting with c2186f0ab928f5cc5fad93fb4332644df2350878a90a56ba54d56eb23c981eab not found: ID does not exist" containerID="c2186f0ab928f5cc5fad93fb4332644df2350878a90a56ba54d56eb23c981eab" Jul 14 19:16:56.784662 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.784544 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2186f0ab928f5cc5fad93fb4332644df2350878a90a56ba54d56eb23c981eab"} err="failed to get container status \"c2186f0ab928f5cc5fad93fb4332644df2350878a90a56ba54d56eb23c981eab\": rpc error: code = NotFound desc = could not find container \"c2186f0ab928f5cc5fad93fb4332644df2350878a90a56ba54d56eb23c981eab\": container with ID starting with c2186f0ab928f5cc5fad93fb4332644df2350878a90a56ba54d56eb23c981eab not found: ID does not exist" Jul 14 19:16:56.784662 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.784566 2576 scope.go:117] "RemoveContainer" containerID="d0263cc23b3526cefce08631ac2c133c3ac911550965a06ea0af63abf1cbe7c5" Jul 14 19:16:56.784938 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:16:56.784919 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0263cc23b3526cefce08631ac2c133c3ac911550965a06ea0af63abf1cbe7c5\": container with ID starting with d0263cc23b3526cefce08631ac2c133c3ac911550965a06ea0af63abf1cbe7c5 not found: ID does not exist" containerID="d0263cc23b3526cefce08631ac2c133c3ac911550965a06ea0af63abf1cbe7c5" Jul 14 19:16:56.785010 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.784941 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0263cc23b3526cefce08631ac2c133c3ac911550965a06ea0af63abf1cbe7c5"} err="failed to get container status \"d0263cc23b3526cefce08631ac2c133c3ac911550965a06ea0af63abf1cbe7c5\": rpc error: code = NotFound desc = could not find container \"d0263cc23b3526cefce08631ac2c133c3ac911550965a06ea0af63abf1cbe7c5\": container with ID starting with d0263cc23b3526cefce08631ac2c133c3ac911550965a06ea0af63abf1cbe7c5 not found: ID does not exist" Jul 14 19:16:56.785010 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.784956 2576 scope.go:117] "RemoveContainer" containerID="863e7a765e4e2bffca749e2d6164389f72eba487396dd0a69ac0475000ad17d6" Jul 14 19:16:56.785345 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:16:56.785320 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"863e7a765e4e2bffca749e2d6164389f72eba487396dd0a69ac0475000ad17d6\": container with ID starting with 863e7a765e4e2bffca749e2d6164389f72eba487396dd0a69ac0475000ad17d6 not found: ID does not exist" containerID="863e7a765e4e2bffca749e2d6164389f72eba487396dd0a69ac0475000ad17d6" Jul 14 19:16:56.785401 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.785354 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863e7a765e4e2bffca749e2d6164389f72eba487396dd0a69ac0475000ad17d6"} err="failed to get container status \"863e7a765e4e2bffca749e2d6164389f72eba487396dd0a69ac0475000ad17d6\": rpc error: code = NotFound desc = could not find container \"863e7a765e4e2bffca749e2d6164389f72eba487396dd0a69ac0475000ad17d6\": container with ID starting with 863e7a765e4e2bffca749e2d6164389f72eba487396dd0a69ac0475000ad17d6 not found: ID does not exist" Jul 14 19:16:56.785401 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.785373 2576 scope.go:117] "RemoveContainer" containerID="337b09272f81bf1bb04a79328d664f54655801cad3b6d09bf7eb4085ba88ab8a" Jul 14 19:16:56.785673 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:16:56.785651 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"337b09272f81bf1bb04a79328d664f54655801cad3b6d09bf7eb4085ba88ab8a\": container with ID starting with 337b09272f81bf1bb04a79328d664f54655801cad3b6d09bf7eb4085ba88ab8a not found: ID does not exist" containerID="337b09272f81bf1bb04a79328d664f54655801cad3b6d09bf7eb4085ba88ab8a" Jul 14 19:16:56.785745 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.785680 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337b09272f81bf1bb04a79328d664f54655801cad3b6d09bf7eb4085ba88ab8a"} err="failed to get container status \"337b09272f81bf1bb04a79328d664f54655801cad3b6d09bf7eb4085ba88ab8a\": rpc error: code = NotFound desc = could not find container \"337b09272f81bf1bb04a79328d664f54655801cad3b6d09bf7eb4085ba88ab8a\": container with ID starting with 337b09272f81bf1bb04a79328d664f54655801cad3b6d09bf7eb4085ba88ab8a not found: ID does not exist" Jul 14 19:16:56.785745 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.785698 2576 scope.go:117] "RemoveContainer" containerID="cb950f5e67fe05a6567940531f8ecd4e6367fdcdaf725e34d065ef00fae25993" Jul 14 19:16:56.785996 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.785978 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb950f5e67fe05a6567940531f8ecd4e6367fdcdaf725e34d065ef00fae25993"} err="failed to get container status \"cb950f5e67fe05a6567940531f8ecd4e6367fdcdaf725e34d065ef00fae25993\": rpc error: code = NotFound desc = could not find container \"cb950f5e67fe05a6567940531f8ecd4e6367fdcdaf725e34d065ef00fae25993\": container with ID starting with cb950f5e67fe05a6567940531f8ecd4e6367fdcdaf725e34d065ef00fae25993 not found: ID does not exist" Jul 14 19:16:56.786046 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.785996 2576 scope.go:117] "RemoveContainer" containerID="4c88039476cf4dfa61f9860f82e8eebd32b8e77dbae97fa909c669d704458480" Jul 14 19:16:56.786238 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.786222 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c88039476cf4dfa61f9860f82e8eebd32b8e77dbae97fa909c669d704458480"} err="failed to get container status \"4c88039476cf4dfa61f9860f82e8eebd32b8e77dbae97fa909c669d704458480\": rpc error: code = NotFound desc = could not find container \"4c88039476cf4dfa61f9860f82e8eebd32b8e77dbae97fa909c669d704458480\": container with ID starting with 4c88039476cf4dfa61f9860f82e8eebd32b8e77dbae97fa909c669d704458480 not found: ID does not exist" Jul 14 19:16:56.786281 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.786236 2576 scope.go:117] "RemoveContainer" containerID="44c71abd667fb0a23324d4eeb06efc776fef03391f291a59a406c9c249a33a72" Jul 14 19:16:56.786467 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.786451 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c71abd667fb0a23324d4eeb06efc776fef03391f291a59a406c9c249a33a72"} err="failed to get container status \"44c71abd667fb0a23324d4eeb06efc776fef03391f291a59a406c9c249a33a72\": rpc error: code = NotFound desc = could not find container \"44c71abd667fb0a23324d4eeb06efc776fef03391f291a59a406c9c249a33a72\": container with ID starting with 44c71abd667fb0a23324d4eeb06efc776fef03391f291a59a406c9c249a33a72 not found: ID does not exist" Jul 14 19:16:56.786467 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.786466 2576 scope.go:117] "RemoveContainer" containerID="c2186f0ab928f5cc5fad93fb4332644df2350878a90a56ba54d56eb23c981eab" Jul 14 19:16:56.786690 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.786669 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2186f0ab928f5cc5fad93fb4332644df2350878a90a56ba54d56eb23c981eab"} err="failed to get container status \"c2186f0ab928f5cc5fad93fb4332644df2350878a90a56ba54d56eb23c981eab\": rpc error: code = NotFound desc = could not find container \"c2186f0ab928f5cc5fad93fb4332644df2350878a90a56ba54d56eb23c981eab\": container with ID starting with c2186f0ab928f5cc5fad93fb4332644df2350878a90a56ba54d56eb23c981eab not found: ID does not exist" Jul 14 19:16:56.786754 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.786691 2576 scope.go:117] "RemoveContainer" containerID="d0263cc23b3526cefce08631ac2c133c3ac911550965a06ea0af63abf1cbe7c5" Jul 14 19:16:56.786894 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.786876 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0263cc23b3526cefce08631ac2c133c3ac911550965a06ea0af63abf1cbe7c5"} err="failed to get container status \"d0263cc23b3526cefce08631ac2c133c3ac911550965a06ea0af63abf1cbe7c5\": rpc error: code = NotFound desc = could not find container \"d0263cc23b3526cefce08631ac2c133c3ac911550965a06ea0af63abf1cbe7c5\": container with ID starting with d0263cc23b3526cefce08631ac2c133c3ac911550965a06ea0af63abf1cbe7c5 not found: ID does not exist" Jul 14 19:16:56.786894 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.786890 2576 scope.go:117] "RemoveContainer" containerID="863e7a765e4e2bffca749e2d6164389f72eba487396dd0a69ac0475000ad17d6" Jul 14 19:16:56.787059 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.787045 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863e7a765e4e2bffca749e2d6164389f72eba487396dd0a69ac0475000ad17d6"} err="failed to get container status \"863e7a765e4e2bffca749e2d6164389f72eba487396dd0a69ac0475000ad17d6\": rpc error: code = NotFound desc = could not find container \"863e7a765e4e2bffca749e2d6164389f72eba487396dd0a69ac0475000ad17d6\": container with ID starting with 863e7a765e4e2bffca749e2d6164389f72eba487396dd0a69ac0475000ad17d6 not found: ID does not exist" Jul 14 19:16:56.787100 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.787060 2576 scope.go:117] "RemoveContainer" containerID="337b09272f81bf1bb04a79328d664f54655801cad3b6d09bf7eb4085ba88ab8a" Jul 14 19:16:56.787243 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:56.787220 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337b09272f81bf1bb04a79328d664f54655801cad3b6d09bf7eb4085ba88ab8a"} err="failed to get container status \"337b09272f81bf1bb04a79328d664f54655801cad3b6d09bf7eb4085ba88ab8a\": rpc error: code = NotFound desc = could not find container \"337b09272f81bf1bb04a79328d664f54655801cad3b6d09bf7eb4085ba88ab8a\": container with ID starting with 337b09272f81bf1bb04a79328d664f54655801cad3b6d09bf7eb4085ba88ab8a not found: ID does not exist" Jul 14 19:16:57.513599 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:57.513560 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" path="/var/lib/kubelet/pods/5b672f03-23ea-450b-83a1-9b6d8f8de338/volumes" Jul 14 19:16:59.544884 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.544844 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 19:16:59.545338 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.545156 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="prometheus" containerID="cri-o://07abb4b61c660e64f2bae6b1841f2e4c4fe8dfabbdb1a3969b35307d1ef41f2a" gracePeriod=600 Jul 14 19:16:59.545338 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.545190 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="kube-rbac-proxy" containerID="cri-o://1316e1ed01f270fd6e9480765b1c99a87b7fa1c3819df2f1624a4b4734f3b64b" gracePeriod=600 Jul 14 19:16:59.545338 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.545228 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="kube-rbac-proxy-web" containerID="cri-o://ca6d57a56e0b77f4fbc12051f9bc1d156cb91266290cc1e3eac137d3bdeaef0a" gracePeriod=600 Jul 14 19:16:59.545338 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.545214 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="thanos-sidecar" containerID="cri-o://28753fdb9c82d1d176bb5f8c3bea514e119773c4d8e77e11c364f0f515b1be4b" gracePeriod=600 Jul 14 19:16:59.545338 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.545236 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="config-reloader" containerID="cri-o://a4a3cb7b89940f91d7ecf1e427c053c004ca70d0fa182e8d2d24942b1e22930d" gracePeriod=600 Jul 14 19:16:59.545338 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.545238 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b15e6116f94841672d982f6a15385a31838ac02e9efbd04449d132f5612bebe4" gracePeriod=600 Jul 14 19:16:59.750258 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.750228 2576 generic.go:356] "Generic (PLEG): container finished" podID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerID="b15e6116f94841672d982f6a15385a31838ac02e9efbd04449d132f5612bebe4" exitCode=0 Jul 14 19:16:59.750258 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.750250 2576 generic.go:356] "Generic (PLEG): container finished" podID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerID="1316e1ed01f270fd6e9480765b1c99a87b7fa1c3819df2f1624a4b4734f3b64b" exitCode=0 Jul 14 19:16:59.750258 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.750256 2576 generic.go:356] "Generic (PLEG): container finished" podID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerID="ca6d57a56e0b77f4fbc12051f9bc1d156cb91266290cc1e3eac137d3bdeaef0a" exitCode=0 Jul 14 19:16:59.750258 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.750263 2576 generic.go:356] "Generic (PLEG): container finished" podID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerID="28753fdb9c82d1d176bb5f8c3bea514e119773c4d8e77e11c364f0f515b1be4b" exitCode=0 Jul 14 19:16:59.750258 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.750269 2576 generic.go:356] "Generic (PLEG): container finished" podID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerID="a4a3cb7b89940f91d7ecf1e427c053c004ca70d0fa182e8d2d24942b1e22930d" exitCode=0 Jul 14 19:16:59.750258 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.750275 2576 generic.go:356] "Generic (PLEG): container finished" podID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerID="07abb4b61c660e64f2bae6b1841f2e4c4fe8dfabbdb1a3969b35307d1ef41f2a" exitCode=0 Jul 14 19:16:59.750710 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.750303 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fb16f4-1dad-47ad-bf02-1269d56f90ac","Type":"ContainerDied","Data":"b15e6116f94841672d982f6a15385a31838ac02e9efbd04449d132f5612bebe4"} Jul 14 19:16:59.750710 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.750375 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fb16f4-1dad-47ad-bf02-1269d56f90ac","Type":"ContainerDied","Data":"1316e1ed01f270fd6e9480765b1c99a87b7fa1c3819df2f1624a4b4734f3b64b"} Jul 14 19:16:59.750710 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.750403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fb16f4-1dad-47ad-bf02-1269d56f90ac","Type":"ContainerDied","Data":"ca6d57a56e0b77f4fbc12051f9bc1d156cb91266290cc1e3eac137d3bdeaef0a"} Jul 14 19:16:59.750710 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.750426 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fb16f4-1dad-47ad-bf02-1269d56f90ac","Type":"ContainerDied","Data":"28753fdb9c82d1d176bb5f8c3bea514e119773c4d8e77e11c364f0f515b1be4b"} Jul 14 19:16:59.750710 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.750450 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fb16f4-1dad-47ad-bf02-1269d56f90ac","Type":"ContainerDied","Data":"a4a3cb7b89940f91d7ecf1e427c053c004ca70d0fa182e8d2d24942b1e22930d"} Jul 14 19:16:59.750710 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.750472 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fb16f4-1dad-47ad-bf02-1269d56f90ac","Type":"ContainerDied","Data":"07abb4b61c660e64f2bae6b1841f2e4c4fe8dfabbdb1a3969b35307d1ef41f2a"} Jul 14 19:16:59.784268 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.784244 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:16:59.888939 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.888904 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7hjj\" (UniqueName: \"kubernetes.io/projected/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-kube-api-access-c7hjj\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.888939 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.888938 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-tls-assets\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889186 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.888968 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889186 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.888992 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-config-out\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889186 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889009 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-rulefiles-1\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889186 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889038 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-configmap-serving-certs-ca-bundle\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889186 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889062 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-metrics-client-certs\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889186 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889111 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-web-config\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889186 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889145 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-prometheus-k8s-tls\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889186 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889167 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-configmap-metrics-client-ca\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889186 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889191 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-rulefiles-2\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889643 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889229 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-configmap-kubelet-serving-ca-bundle\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889643 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889260 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-thanos-prometheus-http-client-file\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889643 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889298 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-config\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889643 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889322 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-grpc-tls\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889643 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889345 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-kube-rbac-proxy\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889643 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889383 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-trusted-ca-bundle\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889643 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889416 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-db\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889643 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889462 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889643 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889479 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:16:59.889643 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889495 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-rulefiles-0\") pod \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\" (UID: \"f9fb16f4-1dad-47ad-bf02-1269d56f90ac\") " Jul 14 19:16:59.889643 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889561 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:16:59.890162 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889688 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:16:59.890162 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889909 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.890162 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889932 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.890162 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.889949 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.890618 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.890557 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:16:59.891101 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.891069 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:16:59.891911 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.891882 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:16:59.892113 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.892081 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:59.892609 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.892568 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:16:59.893092 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.892914 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:16:59.893092 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.892991 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:59.893092 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.893068 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:59.893280 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.893163 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-config-out" (OuterVolumeSpecName: "config-out") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:16:59.893280 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.893189 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:16:59.893374 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.893286 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:59.893518 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.893492 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-kube-api-access-c7hjj" (OuterVolumeSpecName: "kube-api-access-c7hjj") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "kube-api-access-c7hjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:16:59.893611 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.893572 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:59.893934 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.893916 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:59.895113 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.895085 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-config" (OuterVolumeSpecName: "config") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:59.895113 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.895096 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:59.903946 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.903916 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-web-config" (OuterVolumeSpecName: "web-config") pod "f9fb16f4-1dad-47ad-bf02-1269d56f90ac" (UID: "f9fb16f4-1dad-47ad-bf02-1269d56f90ac"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:16:59.991405 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.991365 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.991405 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.991394 2576 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.991405 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.991404 2576 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-config\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.991405 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.991413 2576 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-grpc-tls\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.991676 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.991424 2576 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-kube-rbac-proxy\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.991676 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.991432 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.991676 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.991440 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-db\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.991676 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.991449 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.991676 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.991458 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.991676 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.991466 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7hjj\" (UniqueName: \"kubernetes.io/projected/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-kube-api-access-c7hjj\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.991676 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.991476 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-tls-assets\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.991676 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.991484 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.991676 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.991492 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-config-out\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.991676 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.991500 2576 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-metrics-client-certs\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.991676 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.991508 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-web-config\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.991676 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.991516 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:16:59.991676 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:16:59.991523 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9fb16f4-1dad-47ad-bf02-1269d56f90ac-configmap-metrics-client-ca\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:17:00.756940 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.756911 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fb16f4-1dad-47ad-bf02-1269d56f90ac","Type":"ContainerDied","Data":"9abc27ebf27838da108672e171b60a7b20c39446a84e60841883e4f0ece7055c"} Jul 14 19:17:00.757343 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.756952 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.757343 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.756982 2576 scope.go:117] "RemoveContainer" containerID="b15e6116f94841672d982f6a15385a31838ac02e9efbd04449d132f5612bebe4" Jul 14 19:17:00.765040 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.765016 2576 scope.go:117] "RemoveContainer" containerID="1316e1ed01f270fd6e9480765b1c99a87b7fa1c3819df2f1624a4b4734f3b64b" Jul 14 19:17:00.772492 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.772464 2576 scope.go:117] "RemoveContainer" containerID="ca6d57a56e0b77f4fbc12051f9bc1d156cb91266290cc1e3eac137d3bdeaef0a" Jul 14 19:17:00.778672 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.778656 2576 scope.go:117] "RemoveContainer" containerID="28753fdb9c82d1d176bb5f8c3bea514e119773c4d8e77e11c364f0f515b1be4b" Jul 14 19:17:00.785327 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.785307 2576 scope.go:117] "RemoveContainer" containerID="a4a3cb7b89940f91d7ecf1e427c053c004ca70d0fa182e8d2d24942b1e22930d" Jul 14 19:17:00.786223 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.786200 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 19:17:00.791968 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.791954 2576 scope.go:117] "RemoveContainer" containerID="07abb4b61c660e64f2bae6b1841f2e4c4fe8dfabbdb1a3969b35307d1ef41f2a" Jul 14 19:17:00.798718 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.798696 2576 scope.go:117] "RemoveContainer" containerID="f3075527a8b1f7caff5b3b991bab9f88527b2f9ac93bcf71cd1ab434f8e4ad2b" Jul 14 19:17:00.801023 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.801002 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 19:17:00.840901 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.840874 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 19:17:00.841150 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841138 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="init-config-reloader" Jul 14 19:17:00.841193 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841152 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="init-config-reloader" Jul 14 19:17:00.841193 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841162 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="config-reloader" Jul 14 19:17:00.841193 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841168 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="config-reloader" Jul 14 19:17:00.841193 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841174 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="thanos-sidecar" Jul 14 19:17:00.841193 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841179 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="thanos-sidecar" Jul 14 19:17:00.841193 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841185 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="kube-rbac-proxy-web" Jul 14 19:17:00.841193 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841190 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="kube-rbac-proxy-web" Jul 14 19:17:00.841193 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841196 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="kube-rbac-proxy" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841201 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="kube-rbac-proxy" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841211 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="kube-rbac-proxy" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841216 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="kube-rbac-proxy" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841222 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="config-reloader" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841228 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="config-reloader" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841235 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1e637f7-692d-4334-9ad0-79a5d250479e" containerName="console" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841240 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1e637f7-692d-4334-9ad0-79a5d250479e" containerName="console" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841247 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="kube-rbac-proxy-metric" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841251 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="kube-rbac-proxy-metric" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841256 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="prometheus" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841261 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="prometheus" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841269 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="kube-rbac-proxy-web" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841274 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="kube-rbac-proxy-web" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841280 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="alertmanager" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841284 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="alertmanager" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841292 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="kube-rbac-proxy-thanos" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841297 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="kube-rbac-proxy-thanos" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841303 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="init-config-reloader" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841308 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="init-config-reloader" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841314 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="prom-label-proxy" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841319 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="prom-label-proxy" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841361 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="alertmanager" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841369 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="config-reloader" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841374 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="config-reloader" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841379 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="kube-rbac-proxy" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841385 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="kube-rbac-proxy-web" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841390 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="thanos-sidecar" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841396 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="kube-rbac-proxy-web" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841401 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1e637f7-692d-4334-9ad0-79a5d250479e" containerName="console" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841406 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="kube-rbac-proxy-thanos" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841411 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" containerName="prometheus" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841417 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="kube-rbac-proxy" Jul 14 19:17:00.841410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841422 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="kube-rbac-proxy-metric" Jul 14 19:17:00.842294 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.841428 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b672f03-23ea-450b-83a1-9b6d8f8de338" containerName="prom-label-proxy" Jul 14 19:17:00.845606 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.845575 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.848607 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.848567 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 14 19:17:00.848724 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.848643 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 14 19:17:00.848724 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.848650 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dhjdz\"" Jul 14 19:17:00.848830 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.848739 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 14 19:17:00.848881 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.848867 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 14 19:17:00.848992 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.848978 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fgvakbes7hip8\"" Jul 14 19:17:00.849033 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.848990 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 14 19:17:00.849033 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.849023 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 14 19:17:00.849125 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.849105 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 14 19:17:00.849176 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.849159 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 14 19:17:00.849253 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.849236 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 14 19:17:00.849793 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.849778 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 14 19:17:00.849793 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.849786 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 14 19:17:00.849919 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.849784 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 14 19:17:00.850094 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.850076 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 14 19:17:00.852131 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.852112 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 14 19:17:00.854374 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.854353 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 14 19:17:00.864642 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.864621 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 19:17:00.999168 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999126 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999318 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999318 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999214 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8369730-632e-42e9-9083-fc0027edf7fc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999318 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999231 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999318 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999249 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8369730-632e-42e9-9083-fc0027edf7fc-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999318 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999274 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999318 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999311 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999513 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999327 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999513 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999369 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-config\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999513 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999402 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999513 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999420 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999513 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999438 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8369730-632e-42e9-9083-fc0027edf7fc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999513 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999746 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999525 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999746 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999555 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999746 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999746 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999623 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999746 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999746 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:00.999746 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:00.999693 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92pzf\" (UniqueName: \"kubernetes.io/projected/b8369730-632e-42e9-9083-fc0027edf7fc-kube-api-access-92pzf\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.100178 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.100092 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8369730-632e-42e9-9083-fc0027edf7fc-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.100178 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.100133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.100178 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.100160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.100178 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.100179 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.100488 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.100197 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-config\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.100488 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.100229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.100488 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.100254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.100488 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.100281 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8369730-632e-42e9-9083-fc0027edf7fc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.100721 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.100503 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.100721 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.100695 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8369730-632e-42e9-9083-fc0027edf7fc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.100721 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.100631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.100914 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.100782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.100914 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.100823 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.100914 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.100840 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.100914 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.100865 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.100914 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.100889 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.101929 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.100920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.101929 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.100973 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92pzf\" (UniqueName: \"kubernetes.io/projected/b8369730-632e-42e9-9083-fc0027edf7fc-kube-api-access-92pzf\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.101929 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.101008 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.101929 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.101539 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.101929 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.101873 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.102216 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.102015 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.102216 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.102207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.102364 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.102304 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.103280 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.102487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8369730-632e-42e9-9083-fc0027edf7fc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.103280 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.102526 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.103280 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.103224 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.103719 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.103670 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.104723 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.104696 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8369730-632e-42e9-9083-fc0027edf7fc-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.105483 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.105458 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-config\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.106078 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.105624 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8369730-632e-42e9-9083-fc0027edf7fc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.106078 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.105974 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.106078 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.106026 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.106078 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.106044 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.106078 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.106044 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.106297 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.106131 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8369730-632e-42e9-9083-fc0027edf7fc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.106417 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.106395 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.106498 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.106477 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.106925 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.106909 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8369730-632e-42e9-9083-fc0027edf7fc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.111204 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.111178 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92pzf\" (UniqueName: \"kubernetes.io/projected/b8369730-632e-42e9-9083-fc0027edf7fc-kube-api-access-92pzf\") pod \"prometheus-k8s-0\" (UID: \"b8369730-632e-42e9-9083-fc0027edf7fc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.155890 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.155853 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:17:01.301678 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.301647 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 19:17:01.303848 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.303819 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs\") pod \"network-metrics-daemon-xczln\" (UID: \"8d1706ff-a844-42e9-b6d8-628495d88055\") " pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:17:01.303959 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:17:01.303937 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8369730_632e_42e9_9083_fc0027edf7fc.slice/crio-efdb5532d95a3a0dc146f0ffd431b6b24ba04450aac35129ddcf39a7242c6949 WatchSource:0}: Error finding container efdb5532d95a3a0dc146f0ffd431b6b24ba04450aac35129ddcf39a7242c6949: Status 404 returned error can't find the container with id efdb5532d95a3a0dc146f0ffd431b6b24ba04450aac35129ddcf39a7242c6949 Jul 14 19:17:01.306251 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.306230 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d1706ff-a844-42e9-b6d8-628495d88055-metrics-certs\") pod \"network-metrics-daemon-xczln\" (UID: \"8d1706ff-a844-42e9-b6d8-628495d88055\") " pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:17:01.312351 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.312327 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xxpfz\"" Jul 14 19:17:01.319806 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.319642 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xczln" Jul 14 19:17:01.456397 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.456366 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xczln"] Jul 14 19:17:01.458522 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:17:01.458495 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1706ff_a844_42e9_b6d8_628495d88055.slice/crio-11e9dcd47a938991da4e8fde68e65a879962e79c1cbb45defc517eb94e5b8227 WatchSource:0}: Error finding container 11e9dcd47a938991da4e8fde68e65a879962e79c1cbb45defc517eb94e5b8227: Status 404 returned error can't find the container with id 11e9dcd47a938991da4e8fde68e65a879962e79c1cbb45defc517eb94e5b8227 Jul 14 19:17:01.460427 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.460408 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:17:01.513597 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.513553 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9fb16f4-1dad-47ad-bf02-1269d56f90ac" path="/var/lib/kubelet/pods/f9fb16f4-1dad-47ad-bf02-1269d56f90ac/volumes" Jul 14 19:17:01.762815 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.762731 2576 generic.go:356] "Generic (PLEG): container finished" podID="b8369730-632e-42e9-9083-fc0027edf7fc" containerID="b798c7a888609c67838def98558dc755a729e2b39eba19953447d400a78a38c0" exitCode=0 Jul 14 19:17:01.763219 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.762813 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8369730-632e-42e9-9083-fc0027edf7fc","Type":"ContainerDied","Data":"b798c7a888609c67838def98558dc755a729e2b39eba19953447d400a78a38c0"} Jul 14 19:17:01.763219 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.762885 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8369730-632e-42e9-9083-fc0027edf7fc","Type":"ContainerStarted","Data":"efdb5532d95a3a0dc146f0ffd431b6b24ba04450aac35129ddcf39a7242c6949"} Jul 14 19:17:01.764037 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:01.763931 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xczln" event={"ID":"8d1706ff-a844-42e9-b6d8-628495d88055","Type":"ContainerStarted","Data":"11e9dcd47a938991da4e8fde68e65a879962e79c1cbb45defc517eb94e5b8227"} Jul 14 19:17:02.769380 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:02.769276 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xczln" event={"ID":"8d1706ff-a844-42e9-b6d8-628495d88055","Type":"ContainerStarted","Data":"7131726496dd569c4eb32ea6bb7cd7ea2b20aa258add6562193f7955c5130137"} Jul 14 19:17:02.769380 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:02.769373 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xczln" event={"ID":"8d1706ff-a844-42e9-b6d8-628495d88055","Type":"ContainerStarted","Data":"37e5a4cf44f5d8839e246baeb77100b9e7f906adccb0cf2f21a8e826b1386b90"} Jul 14 19:17:02.772380 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:02.772356 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8369730-632e-42e9-9083-fc0027edf7fc","Type":"ContainerStarted","Data":"f7df1ffbb0fd1f0b06823e79ea0c445e68e616fb1204d24255a85192003b1177"} Jul 14 19:17:02.772493 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:02.772411 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8369730-632e-42e9-9083-fc0027edf7fc","Type":"ContainerStarted","Data":"4e43e92ab0b2794cb7badc74b4a5a4aca9970a2fea63f353906ca0dd301fa3d9"} Jul 14 19:17:02.772493 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:02.772437 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8369730-632e-42e9-9083-fc0027edf7fc","Type":"ContainerStarted","Data":"2ad13707572d933349707d6be4c6bfabcb70e79dd7a89d17168c70cb552d0362"} Jul 14 19:17:02.772493 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:02.772462 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8369730-632e-42e9-9083-fc0027edf7fc","Type":"ContainerStarted","Data":"d01b0e73f081b2d12764155f628a9747abb5058317b34caa77dbf1455d0aaa00"} Jul 14 19:17:02.772493 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:02.772494 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8369730-632e-42e9-9083-fc0027edf7fc","Type":"ContainerStarted","Data":"a0f1d0dc10f75e3bb4658d4fdb3b18c0719ad145aca3ba93c71cebaffcf6dc27"} Jul 14 19:17:02.772721 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:02.772521 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8369730-632e-42e9-9083-fc0027edf7fc","Type":"ContainerStarted","Data":"380029f3d432daefc507baeb4b7b484a18b2b1ec4ff6c058631dc1a61addbba6"} Jul 14 19:17:02.786132 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:02.786082 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xczln" podStartSLOduration=374.833717273 podStartE2EDuration="6m15.786067756s" totalImagesPullingTime="952.350483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:17:02.784376092 +0000 UTC m=+375.897625962" watchObservedRunningTime="2026-07-14 19:17:02.786067756 +0000 UTC m=+375.899317625" Jul 14 19:17:02.813317 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:02.813264 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.78574676 podStartE2EDuration="2.813245998s" totalImagesPullingTime="0s" totalInitContainerRuntime="27.499238ms" isStatefulPod=false podCreationTimestamp="2026-07-14 19:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:17:02.811349748 +0000 UTC m=+375.924599620" watchObservedRunningTime="2026-07-14 19:17:02.813245998 +0000 UTC m=+375.926495893" Jul 14 19:17:06.156161 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:17:06.156126 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:18:01.156401 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:18:01.156356 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:18:01.171712 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:18:01.171685 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:18:02.011708 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:18:02.011680 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:20:47.404538 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:47.404509 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:20:47.405102 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:47.404516 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:20:51.174780 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.174751 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z"] Jul 14 19:20:51.178139 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.178118 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:20:51.182100 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.182077 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 19:20:51.182431 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.182411 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2r59f\"" Jul 14 19:20:51.182611 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.182573 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 19:20:51.182761 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.182748 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 19:20:51.183718 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.183688 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 19:20:51.204946 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.204914 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z"] Jul 14 19:20:51.248196 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.248160 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68nbg\" (UniqueName: \"kubernetes.io/projected/2506a3e9-589c-479e-8b3e-37d879abbf36-kube-api-access-68nbg\") pod \"opendatahub-operator-controller-manager-6f9786d97d-vjr9z\" (UID: \"2506a3e9-589c-479e-8b3e-37d879abbf36\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:20:51.248393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.248203 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2506a3e9-589c-479e-8b3e-37d879abbf36-tmp\") pod \"opendatahub-operator-controller-manager-6f9786d97d-vjr9z\" (UID: \"2506a3e9-589c-479e-8b3e-37d879abbf36\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:20:51.248393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.248222 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2506a3e9-589c-479e-8b3e-37d879abbf36-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f9786d97d-vjr9z\" (UID: \"2506a3e9-589c-479e-8b3e-37d879abbf36\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:20:51.248393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.248313 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2506a3e9-589c-479e-8b3e-37d879abbf36-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f9786d97d-vjr9z\" (UID: \"2506a3e9-589c-479e-8b3e-37d879abbf36\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:20:51.248393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.248387 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2506a3e9-589c-479e-8b3e-37d879abbf36-manifests\") pod \"opendatahub-operator-controller-manager-6f9786d97d-vjr9z\" (UID: \"2506a3e9-589c-479e-8b3e-37d879abbf36\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:20:51.349548 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.349509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2506a3e9-589c-479e-8b3e-37d879abbf36-tmp\") pod \"opendatahub-operator-controller-manager-6f9786d97d-vjr9z\" (UID: \"2506a3e9-589c-479e-8b3e-37d879abbf36\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:20:51.349548 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.349548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2506a3e9-589c-479e-8b3e-37d879abbf36-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f9786d97d-vjr9z\" (UID: \"2506a3e9-589c-479e-8b3e-37d879abbf36\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:20:51.349822 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.349602 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2506a3e9-589c-479e-8b3e-37d879abbf36-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f9786d97d-vjr9z\" (UID: \"2506a3e9-589c-479e-8b3e-37d879abbf36\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:20:51.349822 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.349655 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2506a3e9-589c-479e-8b3e-37d879abbf36-manifests\") pod \"opendatahub-operator-controller-manager-6f9786d97d-vjr9z\" (UID: \"2506a3e9-589c-479e-8b3e-37d879abbf36\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:20:51.349822 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.349706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68nbg\" (UniqueName: \"kubernetes.io/projected/2506a3e9-589c-479e-8b3e-37d879abbf36-kube-api-access-68nbg\") pod \"opendatahub-operator-controller-manager-6f9786d97d-vjr9z\" (UID: \"2506a3e9-589c-479e-8b3e-37d879abbf36\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:20:51.350069 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.350040 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2506a3e9-589c-479e-8b3e-37d879abbf36-manifests\") pod \"opendatahub-operator-controller-manager-6f9786d97d-vjr9z\" (UID: \"2506a3e9-589c-479e-8b3e-37d879abbf36\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:20:51.351867 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.351842 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2506a3e9-589c-479e-8b3e-37d879abbf36-tmp\") pod \"opendatahub-operator-controller-manager-6f9786d97d-vjr9z\" (UID: \"2506a3e9-589c-479e-8b3e-37d879abbf36\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:20:51.351986 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.351966 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2506a3e9-589c-479e-8b3e-37d879abbf36-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f9786d97d-vjr9z\" (UID: \"2506a3e9-589c-479e-8b3e-37d879abbf36\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:20:51.352070 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.352053 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2506a3e9-589c-479e-8b3e-37d879abbf36-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f9786d97d-vjr9z\" (UID: \"2506a3e9-589c-479e-8b3e-37d879abbf36\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:20:51.362413 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.362387 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68nbg\" (UniqueName: \"kubernetes.io/projected/2506a3e9-589c-479e-8b3e-37d879abbf36-kube-api-access-68nbg\") pod \"opendatahub-operator-controller-manager-6f9786d97d-vjr9z\" (UID: \"2506a3e9-589c-479e-8b3e-37d879abbf36\") " pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:20:51.490319 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.490232 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:20:51.667900 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:51.667855 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z"] Jul 14 19:20:51.670335 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:20:51.670307 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2506a3e9_589c_479e_8b3e_37d879abbf36.slice/crio-e55e1474bcebee943ca2fedaae01b2c24b417a0c8acd4ed3e0c54ca04c22e382 WatchSource:0}: Error finding container e55e1474bcebee943ca2fedaae01b2c24b417a0c8acd4ed3e0c54ca04c22e382: Status 404 returned error can't find the container with id e55e1474bcebee943ca2fedaae01b2c24b417a0c8acd4ed3e0c54ca04c22e382 Jul 14 19:20:52.637681 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:52.637635 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" event={"ID":"2506a3e9-589c-479e-8b3e-37d879abbf36","Type":"ContainerStarted","Data":"e55e1474bcebee943ca2fedaae01b2c24b417a0c8acd4ed3e0c54ca04c22e382"} Jul 14 19:20:55.653981 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:55.653944 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" event={"ID":"2506a3e9-589c-479e-8b3e-37d879abbf36","Type":"ContainerStarted","Data":"86cf6380944b339865f9ee09c7ecc420982ad1b13dfd08c179ed2d4089755cca"} Jul 14 19:20:56.659636 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:56.659605 2576 generic.go:356] "Generic (PLEG): container finished" podID="2506a3e9-589c-479e-8b3e-37d879abbf36" containerID="86cf6380944b339865f9ee09c7ecc420982ad1b13dfd08c179ed2d4089755cca" exitCode=0 Jul 14 19:20:56.660018 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:56.659689 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" event={"ID":"2506a3e9-589c-479e-8b3e-37d879abbf36","Type":"ContainerDied","Data":"86cf6380944b339865f9ee09c7ecc420982ad1b13dfd08c179ed2d4089755cca"} Jul 14 19:20:57.666173 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:57.666141 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" event={"ID":"2506a3e9-589c-479e-8b3e-37d879abbf36","Type":"ContainerStarted","Data":"19dc678a25fa7203ca1118dc007b07cffa7d532a457bb979ee9b30670a527cb5"} Jul 14 19:20:57.666651 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:57.666277 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:20:57.688959 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:20:57.688897 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" podStartSLOduration=2.029245589 podStartE2EDuration="6.688872384s" totalImagesPullingTime="3.567597843s" totalInitContainerRuntime="1.092028952s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:20:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:20:57.687141756 +0000 UTC m=+610.800391628" watchObservedRunningTime="2026-07-14 19:20:57.688872384 +0000 UTC m=+610.802122255" Jul 14 19:21:08.672952 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:08.672875 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f9786d97d-vjr9z" Jul 14 19:21:13.930041 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:13.930000 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ggkth"] Jul 14 19:21:13.934410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:13.934388 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:13.938250 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:13.938224 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 14 19:21:13.938371 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:13.938293 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 14 19:21:13.938371 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:13.938324 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 14 19:21:13.938520 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:13.938504 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 14 19:21:13.938631 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:13.938615 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sj9wq\"" Jul 14 19:21:13.950498 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:13.950468 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ggkth"] Jul 14 19:21:14.051429 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.051379 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4lnq\" (UniqueName: \"kubernetes.io/projected/97850f9b-1c14-4631-a4b2-20e3e9b67618-kube-api-access-v4lnq\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.051612 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.051464 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.051612 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.051505 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-token\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.051612 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.051554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97850f9b-1c14-4631-a4b2-20e3e9b67618-local-certs\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.051721 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.051651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97850f9b-1c14-4631-a4b2-20e3e9b67618-cacerts\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.051721 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.051690 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.051721 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.051715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.152385 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.152352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-token\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.152568 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.152389 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97850f9b-1c14-4631-a4b2-20e3e9b67618-local-certs\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.152568 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.152415 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97850f9b-1c14-4631-a4b2-20e3e9b67618-cacerts\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.152568 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.152462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.152568 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.152481 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.152568 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.152500 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4lnq\" (UniqueName: \"kubernetes.io/projected/97850f9b-1c14-4631-a4b2-20e3e9b67618-kube-api-access-v4lnq\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.152904 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.152747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.153313 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.153289 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.155312 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.155284 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97850f9b-1c14-4631-a4b2-20e3e9b67618-local-certs\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.155425 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.155322 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.155489 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.155461 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.155489 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.155476 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97850f9b-1c14-4631-a4b2-20e3e9b67618-cacerts\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.160972 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.160952 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-token\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.161208 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.161191 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4lnq\" (UniqueName: \"kubernetes.io/projected/97850f9b-1c14-4631-a4b2-20e3e9b67618-kube-api-access-v4lnq\") pod \"istiod-openshift-gateway-7574459656-ggkth\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.245399 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.245306 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:14.392362 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.392332 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ggkth"] Jul 14 19:21:14.394289 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:21:14.394257 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97850f9b_1c14_4631_a4b2_20e3e9b67618.slice/crio-61875737f228f5a36831347cc6e4a3ab8b31b4a7cdb8322b52d538d9694a6972 WatchSource:0}: Error finding container 61875737f228f5a36831347cc6e4a3ab8b31b4a7cdb8322b52d538d9694a6972: Status 404 returned error can't find the container with id 61875737f228f5a36831347cc6e4a3ab8b31b4a7cdb8322b52d538d9694a6972 Jul 14 19:21:14.737146 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:14.737116 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" event={"ID":"97850f9b-1c14-4631-a4b2-20e3e9b67618","Type":"ContainerStarted","Data":"61875737f228f5a36831347cc6e4a3ab8b31b4a7cdb8322b52d538d9694a6972"} Jul 14 19:21:17.141340 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:17.141306 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:21:17.141604 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:17.141372 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:21:17.756250 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:17.756216 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" event={"ID":"97850f9b-1c14-4631-a4b2-20e3e9b67618","Type":"ContainerStarted","Data":"f0c3078705818417dd9e5602d73f6a8f0701466739cee4f96e56368aa6da8f56"} Jul 14 19:21:17.756512 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:17.756351 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:17.780552 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:17.780478 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" podStartSLOduration=2.036017308 podStartE2EDuration="4.780454234s" totalImagesPullingTime="2.744436926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:21:17.778080769 +0000 UTC m=+630.891330640" watchObservedRunningTime="2026-07-14 19:21:17.780454234 +0000 UTC m=+630.893704105" Jul 14 19:21:18.762125 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:18.762093 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:19.854364 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:19.854324 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs"] Jul 14 19:21:19.858115 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:19.858082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" Jul 14 19:21:19.862680 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:19.862432 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 14 19:21:19.863670 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:19.863636 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 14 19:21:19.863670 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:19.863636 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 14 19:21:19.863670 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:19.863667 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-q676c\"" Jul 14 19:21:19.863932 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:19.863858 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:21:19.863932 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:19.863912 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 14 19:21:19.872614 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:19.872566 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs"] Jul 14 19:21:20.006815 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:20.006780 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmgn9\" (UniqueName: \"kubernetes.io/projected/c2cbdf74-1983-4fa3-b716-04cbbdf805b4-kube-api-access-dmgn9\") pod \"lws-controller-manager-59cf65bbd6-wnvgs\" (UID: \"c2cbdf74-1983-4fa3-b716-04cbbdf805b4\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" Jul 14 19:21:20.007013 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:20.006847 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2cbdf74-1983-4fa3-b716-04cbbdf805b4-metrics-cert\") pod \"lws-controller-manager-59cf65bbd6-wnvgs\" (UID: \"c2cbdf74-1983-4fa3-b716-04cbbdf805b4\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" Jul 14 19:21:20.007013 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:20.006877 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c2cbdf74-1983-4fa3-b716-04cbbdf805b4-manager-config\") pod \"lws-controller-manager-59cf65bbd6-wnvgs\" (UID: \"c2cbdf74-1983-4fa3-b716-04cbbdf805b4\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" Jul 14 19:21:20.007013 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:20.006977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2cbdf74-1983-4fa3-b716-04cbbdf805b4-cert\") pod \"lws-controller-manager-59cf65bbd6-wnvgs\" (UID: \"c2cbdf74-1983-4fa3-b716-04cbbdf805b4\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" Jul 14 19:21:20.107975 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:20.107888 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2cbdf74-1983-4fa3-b716-04cbbdf805b4-metrics-cert\") pod \"lws-controller-manager-59cf65bbd6-wnvgs\" (UID: \"c2cbdf74-1983-4fa3-b716-04cbbdf805b4\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" Jul 14 19:21:20.107975 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:20.107927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c2cbdf74-1983-4fa3-b716-04cbbdf805b4-manager-config\") pod \"lws-controller-manager-59cf65bbd6-wnvgs\" (UID: \"c2cbdf74-1983-4fa3-b716-04cbbdf805b4\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" Jul 14 19:21:20.108156 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:20.108055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2cbdf74-1983-4fa3-b716-04cbbdf805b4-cert\") pod \"lws-controller-manager-59cf65bbd6-wnvgs\" (UID: \"c2cbdf74-1983-4fa3-b716-04cbbdf805b4\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" Jul 14 19:21:20.108156 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:20.108142 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmgn9\" (UniqueName: \"kubernetes.io/projected/c2cbdf74-1983-4fa3-b716-04cbbdf805b4-kube-api-access-dmgn9\") pod \"lws-controller-manager-59cf65bbd6-wnvgs\" (UID: \"c2cbdf74-1983-4fa3-b716-04cbbdf805b4\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" Jul 14 19:21:20.108551 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:20.108533 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c2cbdf74-1983-4fa3-b716-04cbbdf805b4-manager-config\") pod \"lws-controller-manager-59cf65bbd6-wnvgs\" (UID: \"c2cbdf74-1983-4fa3-b716-04cbbdf805b4\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" Jul 14 19:21:20.110460 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:20.110438 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2cbdf74-1983-4fa3-b716-04cbbdf805b4-metrics-cert\") pod \"lws-controller-manager-59cf65bbd6-wnvgs\" (UID: \"c2cbdf74-1983-4fa3-b716-04cbbdf805b4\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" Jul 14 19:21:20.110564 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:20.110482 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2cbdf74-1983-4fa3-b716-04cbbdf805b4-cert\") pod \"lws-controller-manager-59cf65bbd6-wnvgs\" (UID: \"c2cbdf74-1983-4fa3-b716-04cbbdf805b4\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" Jul 14 19:21:20.125060 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:20.125028 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmgn9\" (UniqueName: \"kubernetes.io/projected/c2cbdf74-1983-4fa3-b716-04cbbdf805b4-kube-api-access-dmgn9\") pod \"lws-controller-manager-59cf65bbd6-wnvgs\" (UID: \"c2cbdf74-1983-4fa3-b716-04cbbdf805b4\") " pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" Jul 14 19:21:20.171097 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:20.171051 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" Jul 14 19:21:20.317469 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:20.317433 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs"] Jul 14 19:21:20.327973 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:21:20.327935 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2cbdf74_1983_4fa3_b716_04cbbdf805b4.slice/crio-2f643cc38c74669ce729e36f6037b0697c587a829f73f11b7b81b9ed8911f329 WatchSource:0}: Error finding container 2f643cc38c74669ce729e36f6037b0697c587a829f73f11b7b81b9ed8911f329: Status 404 returned error can't find the container with id 2f643cc38c74669ce729e36f6037b0697c587a829f73f11b7b81b9ed8911f329 Jul 14 19:21:20.770900 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:20.770871 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" event={"ID":"c2cbdf74-1983-4fa3-b716-04cbbdf805b4","Type":"ContainerStarted","Data":"2f643cc38c74669ce729e36f6037b0697c587a829f73f11b7b81b9ed8911f329"} Jul 14 19:21:25.797260 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:25.797161 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" event={"ID":"c2cbdf74-1983-4fa3-b716-04cbbdf805b4","Type":"ContainerStarted","Data":"b70e90266819688c9d418b706158e6e4834ac2988c4aa66cfa244cde28244c42"} Jul 14 19:21:25.797640 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:25.797329 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" Jul 14 19:21:25.835716 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:25.835662 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" podStartSLOduration=1.7122499439999999 podStartE2EDuration="6.835646241s" totalImagesPullingTime="5.123396297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:21:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:21:25.835486364 +0000 UTC m=+638.948736235" watchObservedRunningTime="2026-07-14 19:21:25.835646241 +0000 UTC m=+638.948896111" Jul 14 19:21:36.803359 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:36.803322 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-59cf65bbd6-wnvgs" Jul 14 19:21:39.578289 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.578246 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ggkth"] Jul 14 19:21:39.579010 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.578426 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" podUID="97850f9b-1c14-4631-a4b2-20e3e9b67618" containerName="discovery" containerID="cri-o://f0c3078705818417dd9e5602d73f6a8f0701466739cee4f96e56368aa6da8f56" gracePeriod=30 Jul 14 19:21:39.823252 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.823229 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:39.860950 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.860871 2576 generic.go:356] "Generic (PLEG): container finished" podID="97850f9b-1c14-4631-a4b2-20e3e9b67618" containerID="f0c3078705818417dd9e5602d73f6a8f0701466739cee4f96e56368aa6da8f56" exitCode=0 Jul 14 19:21:39.860950 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.860917 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" Jul 14 19:21:39.860950 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.860932 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" event={"ID":"97850f9b-1c14-4631-a4b2-20e3e9b67618","Type":"ContainerDied","Data":"f0c3078705818417dd9e5602d73f6a8f0701466739cee4f96e56368aa6da8f56"} Jul 14 19:21:39.861144 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.860978 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" event={"ID":"97850f9b-1c14-4631-a4b2-20e3e9b67618","Type":"ContainerDied","Data":"61875737f228f5a36831347cc6e4a3ab8b31b4a7cdb8322b52d538d9694a6972"} Jul 14 19:21:39.861144 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.861006 2576 scope.go:117] "RemoveContainer" containerID="f0c3078705818417dd9e5602d73f6a8f0701466739cee4f96e56368aa6da8f56" Jul 14 19:21:39.869150 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.869119 2576 scope.go:117] "RemoveContainer" containerID="f0c3078705818417dd9e5602d73f6a8f0701466739cee4f96e56368aa6da8f56" Jul 14 19:21:39.869395 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:21:39.869376 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0c3078705818417dd9e5602d73f6a8f0701466739cee4f96e56368aa6da8f56\": container with ID starting with f0c3078705818417dd9e5602d73f6a8f0701466739cee4f96e56368aa6da8f56 not found: ID does not exist" containerID="f0c3078705818417dd9e5602d73f6a8f0701466739cee4f96e56368aa6da8f56" Jul 14 19:21:39.869462 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.869406 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0c3078705818417dd9e5602d73f6a8f0701466739cee4f96e56368aa6da8f56"} err="failed to get container status \"f0c3078705818417dd9e5602d73f6a8f0701466739cee4f96e56368aa6da8f56\": rpc error: code = NotFound desc = could not find container \"f0c3078705818417dd9e5602d73f6a8f0701466739cee4f96e56368aa6da8f56\": container with ID starting with f0c3078705818417dd9e5602d73f6a8f0701466739cee4f96e56368aa6da8f56 not found: ID does not exist" Jul 14 19:21:39.877729 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.877709 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4lnq\" (UniqueName: \"kubernetes.io/projected/97850f9b-1c14-4631-a4b2-20e3e9b67618-kube-api-access-v4lnq\") pod \"97850f9b-1c14-4631-a4b2-20e3e9b67618\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " Jul 14 19:21:39.877806 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.877740 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-token\") pod \"97850f9b-1c14-4631-a4b2-20e3e9b67618\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " Jul 14 19:21:39.877806 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.877765 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-csr-dns-cert\") pod \"97850f9b-1c14-4631-a4b2-20e3e9b67618\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " Jul 14 19:21:39.877806 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.877787 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-kubeconfig\") pod \"97850f9b-1c14-4631-a4b2-20e3e9b67618\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " Jul 14 19:21:39.877955 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.877816 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-csr-ca-configmap\") pod \"97850f9b-1c14-4631-a4b2-20e3e9b67618\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " Jul 14 19:21:39.877955 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.877852 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97850f9b-1c14-4631-a4b2-20e3e9b67618-cacerts\") pod \"97850f9b-1c14-4631-a4b2-20e3e9b67618\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " Jul 14 19:21:39.877955 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.877887 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97850f9b-1c14-4631-a4b2-20e3e9b67618-local-certs\") pod \"97850f9b-1c14-4631-a4b2-20e3e9b67618\" (UID: \"97850f9b-1c14-4631-a4b2-20e3e9b67618\") " Jul 14 19:21:39.878272 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.878236 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "97850f9b-1c14-4631-a4b2-20e3e9b67618" (UID: "97850f9b-1c14-4631-a4b2-20e3e9b67618"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:21:39.880275 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.880249 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "97850f9b-1c14-4631-a4b2-20e3e9b67618" (UID: "97850f9b-1c14-4631-a4b2-20e3e9b67618"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:21:39.880392 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.880366 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "97850f9b-1c14-4631-a4b2-20e3e9b67618" (UID: "97850f9b-1c14-4631-a4b2-20e3e9b67618"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:21:39.880451 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.880394 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-token" (OuterVolumeSpecName: "istio-token") pod "97850f9b-1c14-4631-a4b2-20e3e9b67618" (UID: "97850f9b-1c14-4631-a4b2-20e3e9b67618"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:21:39.880451 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.880403 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97850f9b-1c14-4631-a4b2-20e3e9b67618-local-certs" (OuterVolumeSpecName: "local-certs") pod "97850f9b-1c14-4631-a4b2-20e3e9b67618" (UID: "97850f9b-1c14-4631-a4b2-20e3e9b67618"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:21:39.880451 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.880414 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97850f9b-1c14-4631-a4b2-20e3e9b67618-cacerts" (OuterVolumeSpecName: "cacerts") pod "97850f9b-1c14-4631-a4b2-20e3e9b67618" (UID: "97850f9b-1c14-4631-a4b2-20e3e9b67618"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:21:39.880639 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.880625 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97850f9b-1c14-4631-a4b2-20e3e9b67618-kube-api-access-v4lnq" (OuterVolumeSpecName: "kube-api-access-v4lnq") pod "97850f9b-1c14-4631-a4b2-20e3e9b67618" (UID: "97850f9b-1c14-4631-a4b2-20e3e9b67618"). InnerVolumeSpecName "kube-api-access-v4lnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:21:39.979307 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.979276 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4lnq\" (UniqueName: \"kubernetes.io/projected/97850f9b-1c14-4631-a4b2-20e3e9b67618-kube-api-access-v4lnq\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:21:39.979307 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.979303 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-token\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:21:39.979307 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.979312 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-csr-dns-cert\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:21:39.979529 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.979320 2576 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-kubeconfig\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:21:39.979529 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.979329 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97850f9b-1c14-4631-a4b2-20e3e9b67618-istio-csr-ca-configmap\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:21:39.979529 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.979337 2576 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97850f9b-1c14-4631-a4b2-20e3e9b67618-cacerts\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:21:39.979529 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:39.979347 2576 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97850f9b-1c14-4631-a4b2-20e3e9b67618-local-certs\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:21:40.184045 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:40.184013 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ggkth"] Jul 14 19:21:40.189479 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:40.189456 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ggkth"] Jul 14 19:21:41.514560 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:41.514528 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97850f9b-1c14-4631-a4b2-20e3e9b67618" path="/var/lib/kubelet/pods/97850f9b-1c14-4631-a4b2-20e3e9b67618/volumes" Jul 14 19:21:44.761450 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:44.761404 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-ggkth container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.24:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 19:21:44.761874 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:21:44.761482 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ggkth" podUID="97850f9b-1c14-4631-a4b2-20e3e9b67618" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.24:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 19:22:13.693255 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:13.693219 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j8fw6"] Jul 14 19:22:13.693737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:13.693540 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97850f9b-1c14-4631-a4b2-20e3e9b67618" containerName="discovery" Jul 14 19:22:13.693737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:13.693550 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="97850f9b-1c14-4631-a4b2-20e3e9b67618" containerName="discovery" Jul 14 19:22:13.693737 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:13.693616 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="97850f9b-1c14-4631-a4b2-20e3e9b67618" containerName="discovery" Jul 14 19:22:13.695707 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:13.695685 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-j8fw6" Jul 14 19:22:13.698280 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:13.698255 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 19:22:13.699513 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:13.699490 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-4xvp7\"" Jul 14 19:22:13.699513 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:13.699492 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 19:22:13.705699 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:13.705670 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j8fw6"] Jul 14 19:22:13.771810 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:13.771774 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr7t6\" (UniqueName: \"kubernetes.io/projected/09cf6ea6-41c7-47fe-9529-a458db36977b-kube-api-access-rr7t6\") pod \"kuadrant-operator-catalog-j8fw6\" (UID: \"09cf6ea6-41c7-47fe-9529-a458db36977b\") " pod="kuadrant-system/kuadrant-operator-catalog-j8fw6" Jul 14 19:22:13.872967 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:13.872926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr7t6\" (UniqueName: \"kubernetes.io/projected/09cf6ea6-41c7-47fe-9529-a458db36977b-kube-api-access-rr7t6\") pod \"kuadrant-operator-catalog-j8fw6\" (UID: \"09cf6ea6-41c7-47fe-9529-a458db36977b\") " pod="kuadrant-system/kuadrant-operator-catalog-j8fw6" Jul 14 19:22:13.881367 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:13.881342 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr7t6\" (UniqueName: \"kubernetes.io/projected/09cf6ea6-41c7-47fe-9529-a458db36977b-kube-api-access-rr7t6\") pod \"kuadrant-operator-catalog-j8fw6\" (UID: \"09cf6ea6-41c7-47fe-9529-a458db36977b\") " pod="kuadrant-system/kuadrant-operator-catalog-j8fw6" Jul 14 19:22:14.007382 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:14.007293 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-j8fw6" Jul 14 19:22:14.056260 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:14.056228 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j8fw6"] Jul 14 19:22:14.135485 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:14.135456 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j8fw6"] Jul 14 19:22:14.137860 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:22:14.137827 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09cf6ea6_41c7_47fe_9529_a458db36977b.slice/crio-3a980c12e44e821a6abb25a27d858c2767d6c76c0ccee2a232d54d3d7b19eb50 WatchSource:0}: Error finding container 3a980c12e44e821a6abb25a27d858c2767d6c76c0ccee2a232d54d3d7b19eb50: Status 404 returned error can't find the container with id 3a980c12e44e821a6abb25a27d858c2767d6c76c0ccee2a232d54d3d7b19eb50 Jul 14 19:22:14.139612 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:14.139592 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:22:14.267667 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:14.267570 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rhsms"] Jul 14 19:22:14.270646 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:14.270629 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rhsms" Jul 14 19:22:14.279338 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:14.279316 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rhsms"] Jul 14 19:22:14.377010 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:14.376973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx5lh\" (UniqueName: \"kubernetes.io/projected/80cda187-571e-49c8-b132-d735ac735b5f-kube-api-access-dx5lh\") pod \"kuadrant-operator-catalog-rhsms\" (UID: \"80cda187-571e-49c8-b132-d735ac735b5f\") " pod="kuadrant-system/kuadrant-operator-catalog-rhsms" Jul 14 19:22:14.477617 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:14.477560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx5lh\" (UniqueName: \"kubernetes.io/projected/80cda187-571e-49c8-b132-d735ac735b5f-kube-api-access-dx5lh\") pod \"kuadrant-operator-catalog-rhsms\" (UID: \"80cda187-571e-49c8-b132-d735ac735b5f\") " pod="kuadrant-system/kuadrant-operator-catalog-rhsms" Jul 14 19:22:14.485694 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:14.485662 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx5lh\" (UniqueName: \"kubernetes.io/projected/80cda187-571e-49c8-b132-d735ac735b5f-kube-api-access-dx5lh\") pod \"kuadrant-operator-catalog-rhsms\" (UID: \"80cda187-571e-49c8-b132-d735ac735b5f\") " pod="kuadrant-system/kuadrant-operator-catalog-rhsms" Jul 14 19:22:14.580963 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:14.580931 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rhsms" Jul 14 19:22:14.709503 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:14.709475 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rhsms"] Jul 14 19:22:14.711993 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:22:14.711957 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80cda187_571e_49c8_b132_d735ac735b5f.slice/crio-aa3580b76fd18ce03fe4e3f7e5362963940abb85a3903ecd4c851d2cb562a6ac WatchSource:0}: Error finding container aa3580b76fd18ce03fe4e3f7e5362963940abb85a3903ecd4c851d2cb562a6ac: Status 404 returned error can't find the container with id aa3580b76fd18ce03fe4e3f7e5362963940abb85a3903ecd4c851d2cb562a6ac Jul 14 19:22:15.018218 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:15.018136 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rhsms" event={"ID":"80cda187-571e-49c8-b132-d735ac735b5f","Type":"ContainerStarted","Data":"aa3580b76fd18ce03fe4e3f7e5362963940abb85a3903ecd4c851d2cb562a6ac"} Jul 14 19:22:15.019387 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:15.019361 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-j8fw6" event={"ID":"09cf6ea6-41c7-47fe-9529-a458db36977b","Type":"ContainerStarted","Data":"3a980c12e44e821a6abb25a27d858c2767d6c76c0ccee2a232d54d3d7b19eb50"} Jul 14 19:22:17.325060 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.325022 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c59d87654-pms5h"] Jul 14 19:22:17.328684 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.328610 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.331451 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.331418 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 14 19:22:17.333814 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.332794 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 14 19:22:17.333814 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.332815 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-s9s5l\"" Jul 14 19:22:17.333814 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.333298 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 14 19:22:17.333814 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.333359 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 14 19:22:17.333814 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.333298 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 14 19:22:17.333814 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.333513 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 14 19:22:17.335776 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.334956 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 14 19:22:17.338833 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.338764 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 14 19:22:17.341460 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.341438 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c59d87654-pms5h"] Jul 14 19:22:17.504061 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.504026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e58163a1-1cfb-469f-8fec-0f421faa3932-service-ca\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.504235 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.504081 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e58163a1-1cfb-469f-8fec-0f421faa3932-console-oauth-config\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.504235 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.504129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xdkw\" (UniqueName: \"kubernetes.io/projected/e58163a1-1cfb-469f-8fec-0f421faa3932-kube-api-access-7xdkw\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.504235 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.504160 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e58163a1-1cfb-469f-8fec-0f421faa3932-console-serving-cert\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.504235 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.504180 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e58163a1-1cfb-469f-8fec-0f421faa3932-oauth-serving-cert\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.504235 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.504212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e58163a1-1cfb-469f-8fec-0f421faa3932-console-config\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.504495 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.504247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e58163a1-1cfb-469f-8fec-0f421faa3932-trusted-ca-bundle\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.605034 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.604947 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e58163a1-1cfb-469f-8fec-0f421faa3932-console-config\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.605034 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.604991 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e58163a1-1cfb-469f-8fec-0f421faa3932-trusted-ca-bundle\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.605256 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.605050 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e58163a1-1cfb-469f-8fec-0f421faa3932-service-ca\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.605256 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.605085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e58163a1-1cfb-469f-8fec-0f421faa3932-console-oauth-config\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.605256 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.605146 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xdkw\" (UniqueName: \"kubernetes.io/projected/e58163a1-1cfb-469f-8fec-0f421faa3932-kube-api-access-7xdkw\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.605256 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.605176 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e58163a1-1cfb-469f-8fec-0f421faa3932-console-serving-cert\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.605256 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.605201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e58163a1-1cfb-469f-8fec-0f421faa3932-oauth-serving-cert\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.605826 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.605789 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e58163a1-1cfb-469f-8fec-0f421faa3932-console-config\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.606029 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.605954 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e58163a1-1cfb-469f-8fec-0f421faa3932-oauth-serving-cert\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.606183 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.606145 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e58163a1-1cfb-469f-8fec-0f421faa3932-trusted-ca-bundle\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.606237 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.606210 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e58163a1-1cfb-469f-8fec-0f421faa3932-service-ca\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.608227 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.608204 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e58163a1-1cfb-469f-8fec-0f421faa3932-console-oauth-config\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.608785 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.608752 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e58163a1-1cfb-469f-8fec-0f421faa3932-console-serving-cert\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.615620 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.615560 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xdkw\" (UniqueName: \"kubernetes.io/projected/e58163a1-1cfb-469f-8fec-0f421faa3932-kube-api-access-7xdkw\") pod \"console-c59d87654-pms5h\" (UID: \"e58163a1-1cfb-469f-8fec-0f421faa3932\") " pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:17.645516 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:17.645480 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:18.208538 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:18.208512 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c59d87654-pms5h"] Jul 14 19:22:18.211022 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:22:18.210993 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58163a1_1cfb_469f_8fec_0f421faa3932.slice/crio-c184e431acc0910cbf7fae51706861f25da10ea7e136b1b2a30befbadd24e5dc WatchSource:0}: Error finding container c184e431acc0910cbf7fae51706861f25da10ea7e136b1b2a30befbadd24e5dc: Status 404 returned error can't find the container with id c184e431acc0910cbf7fae51706861f25da10ea7e136b1b2a30befbadd24e5dc Jul 14 19:22:19.039611 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:19.039548 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-j8fw6" event={"ID":"09cf6ea6-41c7-47fe-9529-a458db36977b","Type":"ContainerStarted","Data":"61b64d729c8b21667bf81ee4d3a5b8c7080b200a97341c5358a52fb9c2336dc6"} Jul 14 19:22:19.040102 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:19.039653 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-j8fw6" podUID="09cf6ea6-41c7-47fe-9529-a458db36977b" containerName="registry-server" containerID="cri-o://61b64d729c8b21667bf81ee4d3a5b8c7080b200a97341c5358a52fb9c2336dc6" gracePeriod=2 Jul 14 19:22:19.041191 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:19.041103 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rhsms" event={"ID":"80cda187-571e-49c8-b132-d735ac735b5f","Type":"ContainerStarted","Data":"68b0e141ebfd70171f74d50dce1b5f9b851e0864027a820f6009b3a37562e23f"} Jul 14 19:22:19.042551 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:19.042523 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c59d87654-pms5h" event={"ID":"e58163a1-1cfb-469f-8fec-0f421faa3932","Type":"ContainerStarted","Data":"438e1bf8d4db297e795257288a1068b7677f28a3e218586dd94866ea19f166f3"} Jul 14 19:22:19.042647 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:19.042572 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c59d87654-pms5h" event={"ID":"e58163a1-1cfb-469f-8fec-0f421faa3932","Type":"ContainerStarted","Data":"c184e431acc0910cbf7fae51706861f25da10ea7e136b1b2a30befbadd24e5dc"} Jul 14 19:22:19.057147 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:19.057108 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-j8fw6" podStartSLOduration=2.05738665 podStartE2EDuration="6.057094552s" totalImagesPullingTime="3.999707902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:22:19.054686454 +0000 UTC m=+692.167936325" watchObservedRunningTime="2026-07-14 19:22:19.057094552 +0000 UTC m=+692.170344422" Jul 14 19:22:19.071569 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:19.071519 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-rhsms" podStartSLOduration=1.641504563 podStartE2EDuration="5.071503671s" totalImagesPullingTime="3.429999108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:22:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:22:19.070214769 +0000 UTC m=+692.183464636" watchObservedRunningTime="2026-07-14 19:22:19.071503671 +0000 UTC m=+692.184753540" Jul 14 19:22:19.089401 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:19.089349 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c59d87654-pms5h" podStartSLOduration=2.089332389 podStartE2EDuration="2.089332389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:22:19.088245176 +0000 UTC m=+692.201495053" watchObservedRunningTime="2026-07-14 19:22:19.089332389 +0000 UTC m=+692.202582258" Jul 14 19:22:19.283086 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:19.283046 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-j8fw6" Jul 14 19:22:19.427314 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:19.427262 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rr7t6\" (UniqueName: \"kubernetes.io/projected/09cf6ea6-41c7-47fe-9529-a458db36977b-kube-api-access-rr7t6\") pod \"09cf6ea6-41c7-47fe-9529-a458db36977b\" (UID: \"09cf6ea6-41c7-47fe-9529-a458db36977b\") " Jul 14 19:22:19.429479 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:19.429447 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09cf6ea6-41c7-47fe-9529-a458db36977b-kube-api-access-rr7t6" (OuterVolumeSpecName: "kube-api-access-rr7t6") pod "09cf6ea6-41c7-47fe-9529-a458db36977b" (UID: "09cf6ea6-41c7-47fe-9529-a458db36977b"). InnerVolumeSpecName "kube-api-access-rr7t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:22:19.528399 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:19.528370 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr7t6\" (UniqueName: \"kubernetes.io/projected/09cf6ea6-41c7-47fe-9529-a458db36977b-kube-api-access-rr7t6\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:22:20.047642 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:20.047602 2576 generic.go:356] "Generic (PLEG): container finished" podID="09cf6ea6-41c7-47fe-9529-a458db36977b" containerID="61b64d729c8b21667bf81ee4d3a5b8c7080b200a97341c5358a52fb9c2336dc6" exitCode=0 Jul 14 19:22:20.048093 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:20.047646 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-j8fw6" Jul 14 19:22:20.048093 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:20.047689 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-j8fw6" event={"ID":"09cf6ea6-41c7-47fe-9529-a458db36977b","Type":"ContainerDied","Data":"61b64d729c8b21667bf81ee4d3a5b8c7080b200a97341c5358a52fb9c2336dc6"} Jul 14 19:22:20.048093 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:20.047741 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-j8fw6" event={"ID":"09cf6ea6-41c7-47fe-9529-a458db36977b","Type":"ContainerDied","Data":"3a980c12e44e821a6abb25a27d858c2767d6c76c0ccee2a232d54d3d7b19eb50"} Jul 14 19:22:20.048093 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:20.047762 2576 scope.go:117] "RemoveContainer" containerID="61b64d729c8b21667bf81ee4d3a5b8c7080b200a97341c5358a52fb9c2336dc6" Jul 14 19:22:20.056707 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:20.056686 2576 scope.go:117] "RemoveContainer" containerID="61b64d729c8b21667bf81ee4d3a5b8c7080b200a97341c5358a52fb9c2336dc6" Jul 14 19:22:20.056975 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:22:20.056953 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b64d729c8b21667bf81ee4d3a5b8c7080b200a97341c5358a52fb9c2336dc6\": container with ID starting with 61b64d729c8b21667bf81ee4d3a5b8c7080b200a97341c5358a52fb9c2336dc6 not found: ID does not exist" containerID="61b64d729c8b21667bf81ee4d3a5b8c7080b200a97341c5358a52fb9c2336dc6" Jul 14 19:22:20.057023 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:20.056984 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b64d729c8b21667bf81ee4d3a5b8c7080b200a97341c5358a52fb9c2336dc6"} err="failed to get container status \"61b64d729c8b21667bf81ee4d3a5b8c7080b200a97341c5358a52fb9c2336dc6\": rpc error: code = NotFound desc = could not find container \"61b64d729c8b21667bf81ee4d3a5b8c7080b200a97341c5358a52fb9c2336dc6\": container with ID starting with 61b64d729c8b21667bf81ee4d3a5b8c7080b200a97341c5358a52fb9c2336dc6 not found: ID does not exist" Jul 14 19:22:20.063793 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:20.063767 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j8fw6"] Jul 14 19:22:20.069811 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:20.069784 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j8fw6"] Jul 14 19:22:21.515222 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:21.515189 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09cf6ea6-41c7-47fe-9529-a458db36977b" path="/var/lib/kubelet/pods/09cf6ea6-41c7-47fe-9529-a458db36977b/volumes" Jul 14 19:22:24.581987 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:24.581941 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-rhsms" Jul 14 19:22:24.582486 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:24.582045 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-rhsms" Jul 14 19:22:24.604668 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:24.604634 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-rhsms" Jul 14 19:22:25.092749 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:25.092723 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-rhsms" Jul 14 19:22:27.646519 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:27.646486 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:27.646890 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:27.646543 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:27.651166 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:27.651142 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:28.086163 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:28.086142 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c59d87654-pms5h" Jul 14 19:22:48.767707 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:48.767669 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg"] Jul 14 19:22:48.768187 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:48.768136 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09cf6ea6-41c7-47fe-9529-a458db36977b" containerName="registry-server" Jul 14 19:22:48.768187 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:48.768157 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="09cf6ea6-41c7-47fe-9529-a458db36977b" containerName="registry-server" Jul 14 19:22:48.768305 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:48.768246 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="09cf6ea6-41c7-47fe-9529-a458db36977b" containerName="registry-server" Jul 14 19:22:48.772404 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:48.772385 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:22:48.780251 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:48.780229 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sncpl\"" Jul 14 19:22:48.792792 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:48.792769 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg"] Jul 14 19:22:48.893680 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:48.893648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hrqz\" (UniqueName: \"kubernetes.io/projected/f7fef8bf-ddc5-4a18-a8fe-8287de561095-kube-api-access-4hrqz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jnkhg\" (UID: \"f7fef8bf-ddc5-4a18-a8fe-8287de561095\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:22:48.893826 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:48.893765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7fef8bf-ddc5-4a18-a8fe-8287de561095-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jnkhg\" (UID: \"f7fef8bf-ddc5-4a18-a8fe-8287de561095\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:22:48.994746 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:48.994702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7fef8bf-ddc5-4a18-a8fe-8287de561095-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jnkhg\" (UID: \"f7fef8bf-ddc5-4a18-a8fe-8287de561095\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:22:48.994927 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:48.994767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hrqz\" (UniqueName: \"kubernetes.io/projected/f7fef8bf-ddc5-4a18-a8fe-8287de561095-kube-api-access-4hrqz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jnkhg\" (UID: \"f7fef8bf-ddc5-4a18-a8fe-8287de561095\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:22:48.995109 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:48.995088 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7fef8bf-ddc5-4a18-a8fe-8287de561095-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jnkhg\" (UID: \"f7fef8bf-ddc5-4a18-a8fe-8287de561095\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:22:49.003568 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:49.003540 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hrqz\" (UniqueName: \"kubernetes.io/projected/f7fef8bf-ddc5-4a18-a8fe-8287de561095-kube-api-access-4hrqz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jnkhg\" (UID: \"f7fef8bf-ddc5-4a18-a8fe-8287de561095\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:22:49.082232 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:49.082203 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:22:49.231540 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:49.231501 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg"] Jul 14 19:22:49.235112 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:22:49.235080 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fef8bf_ddc5_4a18_a8fe_8287de561095.slice/crio-4a561bb569375fde1c4959535dd6c8b8fc7603ed748e5a3399abe94a26824580 WatchSource:0}: Error finding container 4a561bb569375fde1c4959535dd6c8b8fc7603ed748e5a3399abe94a26824580: Status 404 returned error can't find the container with id 4a561bb569375fde1c4959535dd6c8b8fc7603ed748e5a3399abe94a26824580 Jul 14 19:22:50.176079 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:50.176042 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" event={"ID":"f7fef8bf-ddc5-4a18-a8fe-8287de561095","Type":"ContainerStarted","Data":"4a561bb569375fde1c4959535dd6c8b8fc7603ed748e5a3399abe94a26824580"} Jul 14 19:22:50.730547 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:50.730511 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr"] Jul 14 19:22:50.734269 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:50.734247 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" Jul 14 19:22:50.737192 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:50.737175 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-zv7fz\"" Jul 14 19:22:50.742622 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:50.742600 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr"] Jul 14 19:22:50.811492 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:50.811449 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4g9v\" (UniqueName: \"kubernetes.io/projected/9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6-kube-api-access-f4g9v\") pod \"limitador-operator-controller-manager-859d7b78bd-xrbqr\" (UID: \"9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" Jul 14 19:22:50.912880 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:50.912839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4g9v\" (UniqueName: \"kubernetes.io/projected/9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6-kube-api-access-f4g9v\") pod \"limitador-operator-controller-manager-859d7b78bd-xrbqr\" (UID: \"9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" Jul 14 19:22:50.924213 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:50.924185 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4g9v\" (UniqueName: \"kubernetes.io/projected/9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6-kube-api-access-f4g9v\") pod \"limitador-operator-controller-manager-859d7b78bd-xrbqr\" (UID: \"9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" Jul 14 19:22:51.046393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:51.046305 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" Jul 14 19:22:51.202194 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:51.202163 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr"] Jul 14 19:22:51.203680 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:22:51.203650 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eefafbf_c9b8_45fd_a5f2_8ce1239c3ad6.slice/crio-00203ed24ee64a34a6d86aa66d474da981a4bc880f20a39deeeffb40606c2394 WatchSource:0}: Error finding container 00203ed24ee64a34a6d86aa66d474da981a4bc880f20a39deeeffb40606c2394: Status 404 returned error can't find the container with id 00203ed24ee64a34a6d86aa66d474da981a4bc880f20a39deeeffb40606c2394 Jul 14 19:22:52.187769 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:52.187733 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" event={"ID":"9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6","Type":"ContainerStarted","Data":"00203ed24ee64a34a6d86aa66d474da981a4bc880f20a39deeeffb40606c2394"} Jul 14 19:22:53.195619 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:53.195552 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" event={"ID":"9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6","Type":"ContainerStarted","Data":"ec2e6eac606d1d5d83df338d19db38e90a71e12df54365314795dd247709c231"} Jul 14 19:22:53.196230 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:53.196194 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" Jul 14 19:22:53.216067 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:53.216003 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" podStartSLOduration=1.424055991 podStartE2EDuration="3.215983551s" totalImagesPullingTime="1.79192756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:22:53.214831998 +0000 UTC m=+726.328081869" watchObservedRunningTime="2026-07-14 19:22:53.215983551 +0000 UTC m=+726.329233421" Jul 14 19:22:56.212489 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:56.212454 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" event={"ID":"f7fef8bf-ddc5-4a18-a8fe-8287de561095","Type":"ContainerStarted","Data":"57bca33113c4be5ed0b3bd6c01746596d8aeb741b374a30e1e3dfde44248202e"} Jul 14 19:22:56.212903 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:56.212570 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:22:56.234021 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:56.233971 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" podStartSLOduration=2.179847073 podStartE2EDuration="8.233954601s" totalImagesPullingTime="6.054107528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:22:56.231893236 +0000 UTC m=+729.345143108" watchObservedRunningTime="2026-07-14 19:22:56.233954601 +0000 UTC m=+729.347204470" Jul 14 19:22:56.811158 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:56.808542 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7btf"] Jul 14 19:22:56.813495 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:56.813465 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7btf" Jul 14 19:22:56.816983 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:56.816955 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 14 19:22:56.822745 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:56.822723 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-hzdzc\"" Jul 14 19:22:56.847893 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:56.847858 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7btf"] Jul 14 19:22:56.964341 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:56.964303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw6kq\" (UniqueName: \"kubernetes.io/projected/5a960dcd-05a0-4e66-af82-ee4f975182e2-kube-api-access-bw6kq\") pod \"dns-operator-controller-manager-864dd78c4c-m7btf\" (UID: \"5a960dcd-05a0-4e66-af82-ee4f975182e2\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7btf" Jul 14 19:22:57.065324 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:57.065230 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw6kq\" (UniqueName: \"kubernetes.io/projected/5a960dcd-05a0-4e66-af82-ee4f975182e2-kube-api-access-bw6kq\") pod \"dns-operator-controller-manager-864dd78c4c-m7btf\" (UID: \"5a960dcd-05a0-4e66-af82-ee4f975182e2\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7btf" Jul 14 19:22:57.074264 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:57.074235 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw6kq\" (UniqueName: \"kubernetes.io/projected/5a960dcd-05a0-4e66-af82-ee4f975182e2-kube-api-access-bw6kq\") pod \"dns-operator-controller-manager-864dd78c4c-m7btf\" (UID: \"5a960dcd-05a0-4e66-af82-ee4f975182e2\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7btf" Jul 14 19:22:57.123932 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:57.123892 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7btf" Jul 14 19:22:57.276871 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:57.276812 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7btf"] Jul 14 19:22:57.279681 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:22:57.279643 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a960dcd_05a0_4e66_af82_ee4f975182e2.slice/crio-b133aafd1698bef889a81d1750bbf2993c8fd47ad7e54f2ddc3fb98d25253051 WatchSource:0}: Error finding container b133aafd1698bef889a81d1750bbf2993c8fd47ad7e54f2ddc3fb98d25253051: Status 404 returned error can't find the container with id b133aafd1698bef889a81d1750bbf2993c8fd47ad7e54f2ddc3fb98d25253051 Jul 14 19:22:58.190979 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:58.190943 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xjwwz"] Jul 14 19:22:58.195078 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:58.195047 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xjwwz" Jul 14 19:22:58.197788 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:58.197764 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 14 19:22:58.197907 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:58.197770 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 14 19:22:58.198535 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:58.198518 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jwxt4\"" Jul 14 19:22:58.203697 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:58.203669 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xjwwz"] Jul 14 19:22:58.224632 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:58.224602 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7btf" event={"ID":"5a960dcd-05a0-4e66-af82-ee4f975182e2","Type":"ContainerStarted","Data":"b133aafd1698bef889a81d1750bbf2993c8fd47ad7e54f2ddc3fb98d25253051"} Jul 14 19:22:58.277121 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:58.277091 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qqmq\" (UniqueName: \"kubernetes.io/projected/bc50cefe-792d-4e62-bb65-bb493dce3b7f-kube-api-access-9qqmq\") pod \"kuadrant-console-plugin-66b5774985-xjwwz\" (UID: \"bc50cefe-792d-4e62-bb65-bb493dce3b7f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xjwwz" Jul 14 19:22:58.277505 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:58.277133 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc50cefe-792d-4e62-bb65-bb493dce3b7f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xjwwz\" (UID: \"bc50cefe-792d-4e62-bb65-bb493dce3b7f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xjwwz" Jul 14 19:22:58.277505 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:58.277265 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc50cefe-792d-4e62-bb65-bb493dce3b7f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xjwwz\" (UID: \"bc50cefe-792d-4e62-bb65-bb493dce3b7f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xjwwz" Jul 14 19:22:58.378450 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:58.378418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc50cefe-792d-4e62-bb65-bb493dce3b7f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xjwwz\" (UID: \"bc50cefe-792d-4e62-bb65-bb493dce3b7f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xjwwz" Jul 14 19:22:58.378648 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:58.378506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qqmq\" (UniqueName: \"kubernetes.io/projected/bc50cefe-792d-4e62-bb65-bb493dce3b7f-kube-api-access-9qqmq\") pod \"kuadrant-console-plugin-66b5774985-xjwwz\" (UID: \"bc50cefe-792d-4e62-bb65-bb493dce3b7f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xjwwz" Jul 14 19:22:58.378648 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:58.378545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc50cefe-792d-4e62-bb65-bb493dce3b7f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xjwwz\" (UID: \"bc50cefe-792d-4e62-bb65-bb493dce3b7f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xjwwz" Jul 14 19:22:58.379431 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:58.379402 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc50cefe-792d-4e62-bb65-bb493dce3b7f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xjwwz\" (UID: \"bc50cefe-792d-4e62-bb65-bb493dce3b7f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xjwwz" Jul 14 19:22:58.381544 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:58.381079 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc50cefe-792d-4e62-bb65-bb493dce3b7f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xjwwz\" (UID: \"bc50cefe-792d-4e62-bb65-bb493dce3b7f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xjwwz" Jul 14 19:22:58.387535 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:58.387513 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qqmq\" (UniqueName: \"kubernetes.io/projected/bc50cefe-792d-4e62-bb65-bb493dce3b7f-kube-api-access-9qqmq\") pod \"kuadrant-console-plugin-66b5774985-xjwwz\" (UID: \"bc50cefe-792d-4e62-bb65-bb493dce3b7f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xjwwz" Jul 14 19:22:58.514943 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:58.514859 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xjwwz" Jul 14 19:22:58.736983 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:58.736957 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xjwwz"] Jul 14 19:22:58.740203 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:22:58.740174 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc50cefe_792d_4e62_bb65_bb493dce3b7f.slice/crio-1f5cd9f0185cbb2e578f5476f6a59bdda9bf05a1b9bdba3a100f5edeb32a88de WatchSource:0}: Error finding container 1f5cd9f0185cbb2e578f5476f6a59bdda9bf05a1b9bdba3a100f5edeb32a88de: Status 404 returned error can't find the container with id 1f5cd9f0185cbb2e578f5476f6a59bdda9bf05a1b9bdba3a100f5edeb32a88de Jul 14 19:22:59.230736 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:22:59.230695 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xjwwz" event={"ID":"bc50cefe-792d-4e62-bb65-bb493dce3b7f","Type":"ContainerStarted","Data":"1f5cd9f0185cbb2e578f5476f6a59bdda9bf05a1b9bdba3a100f5edeb32a88de"} Jul 14 19:23:00.236331 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:00.236290 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7btf" event={"ID":"5a960dcd-05a0-4e66-af82-ee4f975182e2","Type":"ContainerStarted","Data":"49feab31e1d6c3cf86652329527811ea7c571aa7e92d8e460fee63cb5b75e1a6"} Jul 14 19:23:00.236813 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:00.236492 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7btf" Jul 14 19:23:00.255140 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:00.255099 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7btf" podStartSLOduration=1.9467052649999999 podStartE2EDuration="4.255082968s" totalImagesPullingTime="2.308377703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:23:00.254403086 +0000 UTC m=+733.367652957" watchObservedRunningTime="2026-07-14 19:23:00.255082968 +0000 UTC m=+733.368332818" Jul 14 19:23:04.203623 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:04.203569 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" Jul 14 19:23:07.221312 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:07.221282 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:23:08.512886 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:08.512846 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t"] Jul 14 19:23:08.518001 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:08.517975 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t" Jul 14 19:23:08.530099 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:08.530067 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t"] Jul 14 19:23:08.679801 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:08.679767 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxrkp\" (UniqueName: \"kubernetes.io/projected/6e344d1a-88ef-45fb-8a0a-2e2b4a36513f-kube-api-access-pxrkp\") pod \"kuadrant-operator-controller-manager-55467d9f7-5n47t\" (UID: \"6e344d1a-88ef-45fb-8a0a-2e2b4a36513f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t" Jul 14 19:23:08.679952 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:08.679860 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e344d1a-88ef-45fb-8a0a-2e2b4a36513f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-5n47t\" (UID: \"6e344d1a-88ef-45fb-8a0a-2e2b4a36513f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t" Jul 14 19:23:08.780693 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:08.780612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxrkp\" (UniqueName: \"kubernetes.io/projected/6e344d1a-88ef-45fb-8a0a-2e2b4a36513f-kube-api-access-pxrkp\") pod \"kuadrant-operator-controller-manager-55467d9f7-5n47t\" (UID: \"6e344d1a-88ef-45fb-8a0a-2e2b4a36513f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t" Jul 14 19:23:08.780848 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:08.780722 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e344d1a-88ef-45fb-8a0a-2e2b4a36513f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-5n47t\" (UID: \"6e344d1a-88ef-45fb-8a0a-2e2b4a36513f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t" Jul 14 19:23:08.781146 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:08.781125 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e344d1a-88ef-45fb-8a0a-2e2b4a36513f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-5n47t\" (UID: \"6e344d1a-88ef-45fb-8a0a-2e2b4a36513f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t" Jul 14 19:23:08.792011 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:08.791979 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxrkp\" (UniqueName: \"kubernetes.io/projected/6e344d1a-88ef-45fb-8a0a-2e2b4a36513f-kube-api-access-pxrkp\") pod \"kuadrant-operator-controller-manager-55467d9f7-5n47t\" (UID: \"6e344d1a-88ef-45fb-8a0a-2e2b4a36513f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t" Jul 14 19:23:08.830221 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:08.830179 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t" Jul 14 19:23:08.978487 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:08.978445 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t"] Jul 14 19:23:08.988300 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:08.988271 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t"] Jul 14 19:23:09.004285 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.003881 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg"] Jul 14 19:23:09.004285 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.004132 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" podUID="f7fef8bf-ddc5-4a18-a8fe-8287de561095" containerName="manager" containerID="cri-o://57bca33113c4be5ed0b3bd6c01746596d8aeb741b374a30e1e3dfde44248202e" gracePeriod=2 Jul 14 19:23:09.006472 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.006442 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh"] Jul 14 19:23:09.007893 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:09.007833 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jnkhg\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="f7fef8bf-ddc5-4a18-a8fe-8287de561095" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:23:09.010880 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.010848 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg"] Jul 14 19:23:09.011024 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.010978 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh" Jul 14 19:23:09.013517 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:09.013253 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jnkhg\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="f7fef8bf-ddc5-4a18-a8fe-8287de561095" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:23:09.021515 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.021480 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh"] Jul 14 19:23:09.025982 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.025930 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr"] Jul 14 19:23:09.026208 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.026166 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" podUID="9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6" containerName="manager" containerID="cri-o://ec2e6eac606d1d5d83df338d19db38e90a71e12df54365314795dd247709c231" gracePeriod=2 Jul 14 19:23:09.031143 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.031043 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr"] Jul 14 19:23:09.045710 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:09.045673 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jnkhg\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="f7fef8bf-ddc5-4a18-a8fe-8287de561095" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:23:09.047852 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:09.047810 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xrbqr\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" Jul 14 19:23:09.059256 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.059232 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m55b7"] Jul 14 19:23:09.059797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.059614 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6" containerName="manager" Jul 14 19:23:09.059797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.059628 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6" containerName="manager" Jul 14 19:23:09.059797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.059666 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7fef8bf-ddc5-4a18-a8fe-8287de561095" containerName="manager" Jul 14 19:23:09.059797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.059674 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7fef8bf-ddc5-4a18-a8fe-8287de561095" containerName="manager" Jul 14 19:23:09.059797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.059732 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6" containerName="manager" Jul 14 19:23:09.059797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.059743 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7fef8bf-ddc5-4a18-a8fe-8287de561095" containerName="manager" Jul 14 19:23:09.062926 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.062898 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m55b7" Jul 14 19:23:09.079095 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.079057 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m55b7"] Jul 14 19:23:09.098820 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:09.098736 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jnkhg\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="f7fef8bf-ddc5-4a18-a8fe-8287de561095" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:23:09.101151 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:09.101122 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xrbqr\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" Jul 14 19:23:09.184758 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.184695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc228e4b-7949-4456-b0bc-079aeb134472-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-7gvnh\" (UID: \"bc228e4b-7949-4456-b0bc-079aeb134472\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh" Jul 14 19:23:09.185005 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.184765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qcrm\" (UniqueName: \"kubernetes.io/projected/bc228e4b-7949-4456-b0bc-079aeb134472-kube-api-access-8qcrm\") pod \"kuadrant-operator-controller-manager-55467d9f7-7gvnh\" (UID: \"bc228e4b-7949-4456-b0bc-079aeb134472\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh" Jul 14 19:23:09.185005 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.184799 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx26m\" (UniqueName: \"kubernetes.io/projected/a36da07c-6fdc-4ee1-afa9-27e6ac38396b-kube-api-access-fx26m\") pod \"limitador-operator-controller-manager-859d7b78bd-m55b7\" (UID: \"a36da07c-6fdc-4ee1-afa9-27e6ac38396b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m55b7" Jul 14 19:23:09.285852 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.285317 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc228e4b-7949-4456-b0bc-079aeb134472-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-7gvnh\" (UID: \"bc228e4b-7949-4456-b0bc-079aeb134472\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh" Jul 14 19:23:09.285852 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.285378 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qcrm\" (UniqueName: \"kubernetes.io/projected/bc228e4b-7949-4456-b0bc-079aeb134472-kube-api-access-8qcrm\") pod \"kuadrant-operator-controller-manager-55467d9f7-7gvnh\" (UID: \"bc228e4b-7949-4456-b0bc-079aeb134472\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh" Jul 14 19:23:09.285852 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.285715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx26m\" (UniqueName: \"kubernetes.io/projected/a36da07c-6fdc-4ee1-afa9-27e6ac38396b-kube-api-access-fx26m\") pod \"limitador-operator-controller-manager-859d7b78bd-m55b7\" (UID: \"a36da07c-6fdc-4ee1-afa9-27e6ac38396b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m55b7" Jul 14 19:23:09.285852 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.285792 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc228e4b-7949-4456-b0bc-079aeb134472-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-7gvnh\" (UID: \"bc228e4b-7949-4456-b0bc-079aeb134472\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh" Jul 14 19:23:09.312074 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.312030 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qcrm\" (UniqueName: \"kubernetes.io/projected/bc228e4b-7949-4456-b0bc-079aeb134472-kube-api-access-8qcrm\") pod \"kuadrant-operator-controller-manager-55467d9f7-7gvnh\" (UID: \"bc228e4b-7949-4456-b0bc-079aeb134472\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh" Jul 14 19:23:09.314967 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.314942 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx26m\" (UniqueName: \"kubernetes.io/projected/a36da07c-6fdc-4ee1-afa9-27e6ac38396b-kube-api-access-fx26m\") pod \"limitador-operator-controller-manager-859d7b78bd-m55b7\" (UID: \"a36da07c-6fdc-4ee1-afa9-27e6ac38396b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m55b7" Jul 14 19:23:09.404423 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.404389 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh" Jul 14 19:23:09.412461 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:09.412433 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m55b7" Jul 14 19:23:11.245195 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:11.245156 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m7btf" Jul 14 19:23:23.058707 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.058574 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" Jul 14 19:23:23.061239 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:23.061201 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xrbqr\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" Jul 14 19:23:23.062302 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.062245 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:23:23.064669 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:23.064637 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xrbqr\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" Jul 14 19:23:23.066876 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:23.066848 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jnkhg\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="f7fef8bf-ddc5-4a18-a8fe-8287de561095" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:23:23.073177 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.073143 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m55b7"] Jul 14 19:23:23.076489 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:23:23.076444 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36da07c_6fdc_4ee1_afa9_27e6ac38396b.slice/crio-b4ee0430b458457d63ec7da3c1ed582ac747df3483f487101c4175fa43e7aa60 WatchSource:0}: Error finding container b4ee0430b458457d63ec7da3c1ed582ac747df3483f487101c4175fa43e7aa60: Status 404 returned error can't find the container with id b4ee0430b458457d63ec7da3c1ed582ac747df3483f487101c4175fa43e7aa60 Jul 14 19:23:23.104318 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.104291 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh"] Jul 14 19:23:23.121600 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.121558 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hrqz\" (UniqueName: \"kubernetes.io/projected/f7fef8bf-ddc5-4a18-a8fe-8287de561095-kube-api-access-4hrqz\") pod \"f7fef8bf-ddc5-4a18-a8fe-8287de561095\" (UID: \"f7fef8bf-ddc5-4a18-a8fe-8287de561095\") " Jul 14 19:23:23.121744 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.121720 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7fef8bf-ddc5-4a18-a8fe-8287de561095-extensions-socket-volume\") pod \"f7fef8bf-ddc5-4a18-a8fe-8287de561095\" (UID: \"f7fef8bf-ddc5-4a18-a8fe-8287de561095\") " Jul 14 19:23:23.121809 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.121755 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4g9v\" (UniqueName: \"kubernetes.io/projected/9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6-kube-api-access-f4g9v\") pod \"9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6\" (UID: \"9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6\") " Jul 14 19:23:23.122118 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:23:23.122088 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc228e4b_7949_4456_b0bc_079aeb134472.slice/crio-f347fc6140385071b80e5264035a1a0bd8660f0b9e15015964cb1f71a0437161 WatchSource:0}: Error finding container f347fc6140385071b80e5264035a1a0bd8660f0b9e15015964cb1f71a0437161: Status 404 returned error can't find the container with id f347fc6140385071b80e5264035a1a0bd8660f0b9e15015964cb1f71a0437161 Jul 14 19:23:23.122379 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.122342 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7fef8bf-ddc5-4a18-a8fe-8287de561095-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f7fef8bf-ddc5-4a18-a8fe-8287de561095" (UID: "f7fef8bf-ddc5-4a18-a8fe-8287de561095"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:23:23.124124 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.124097 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7fef8bf-ddc5-4a18-a8fe-8287de561095-kube-api-access-4hrqz" (OuterVolumeSpecName: "kube-api-access-4hrqz") pod "f7fef8bf-ddc5-4a18-a8fe-8287de561095" (UID: "f7fef8bf-ddc5-4a18-a8fe-8287de561095"). InnerVolumeSpecName "kube-api-access-4hrqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:23:23.124366 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.124336 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6-kube-api-access-f4g9v" (OuterVolumeSpecName: "kube-api-access-f4g9v") pod "9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6" (UID: "9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6"). InnerVolumeSpecName "kube-api-access-f4g9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:23:23.222820 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.222784 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7fef8bf-ddc5-4a18-a8fe-8287de561095-extensions-socket-volume\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:23:23.222820 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.222818 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4g9v\" (UniqueName: \"kubernetes.io/projected/9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6-kube-api-access-f4g9v\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:23:23.223022 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.222829 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hrqz\" (UniqueName: \"kubernetes.io/projected/f7fef8bf-ddc5-4a18-a8fe-8287de561095-kube-api-access-4hrqz\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:23:23.358681 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.358642 2576 generic.go:356] "Generic (PLEG): container finished" podID="f7fef8bf-ddc5-4a18-a8fe-8287de561095" containerID="57bca33113c4be5ed0b3bd6c01746596d8aeb741b374a30e1e3dfde44248202e" exitCode=0 Jul 14 19:23:23.358878 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.358703 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:23:23.358878 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.358742 2576 scope.go:117] "RemoveContainer" containerID="57bca33113c4be5ed0b3bd6c01746596d8aeb741b374a30e1e3dfde44248202e" Jul 14 19:23:23.360225 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.360200 2576 generic.go:356] "Generic (PLEG): container finished" podID="9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6" containerID="ec2e6eac606d1d5d83df338d19db38e90a71e12df54365314795dd247709c231" exitCode=0 Jul 14 19:23:23.360331 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.360282 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" Jul 14 19:23:23.361948 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:23.361803 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xrbqr\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" Jul 14 19:23:23.361948 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.361828 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xjwwz" event={"ID":"bc50cefe-792d-4e62-bb65-bb493dce3b7f","Type":"ContainerStarted","Data":"7a2fa37b6028d30f2dda99b7ab90e84ab3d694aaa6dfc4771191159432a35687"} Jul 14 19:23:23.363519 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.363497 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m55b7" event={"ID":"a36da07c-6fdc-4ee1-afa9-27e6ac38396b","Type":"ContainerStarted","Data":"4b1e18b53daf162a0b7396bed4f38976a1734611b043dc90253931746c9e3acd"} Jul 14 19:23:23.363654 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.363632 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m55b7" event={"ID":"a36da07c-6fdc-4ee1-afa9-27e6ac38396b","Type":"ContainerStarted","Data":"b4ee0430b458457d63ec7da3c1ed582ac747df3483f487101c4175fa43e7aa60"} Jul 14 19:23:23.363723 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.363669 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m55b7" Jul 14 19:23:23.364251 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:23.364223 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jnkhg\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="f7fef8bf-ddc5-4a18-a8fe-8287de561095" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:23:23.365218 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.365198 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh" event={"ID":"bc228e4b-7949-4456-b0bc-079aeb134472","Type":"ContainerStarted","Data":"4e6dbc018b5220af57f642f1dda1e98565c16c4ddf914e2bc10c7a942ce9e56c"} Jul 14 19:23:23.365310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.365237 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh" event={"ID":"bc228e4b-7949-4456-b0bc-079aeb134472","Type":"ContainerStarted","Data":"f347fc6140385071b80e5264035a1a0bd8660f0b9e15015964cb1f71a0437161"} Jul 14 19:23:23.365310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.365278 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh" Jul 14 19:23:23.368867 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.368847 2576 scope.go:117] "RemoveContainer" containerID="57bca33113c4be5ed0b3bd6c01746596d8aeb741b374a30e1e3dfde44248202e" Jul 14 19:23:23.369138 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:23.369118 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57bca33113c4be5ed0b3bd6c01746596d8aeb741b374a30e1e3dfde44248202e\": container with ID starting with 57bca33113c4be5ed0b3bd6c01746596d8aeb741b374a30e1e3dfde44248202e not found: ID does not exist" containerID="57bca33113c4be5ed0b3bd6c01746596d8aeb741b374a30e1e3dfde44248202e" Jul 14 19:23:23.369215 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.369144 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57bca33113c4be5ed0b3bd6c01746596d8aeb741b374a30e1e3dfde44248202e"} err="failed to get container status \"57bca33113c4be5ed0b3bd6c01746596d8aeb741b374a30e1e3dfde44248202e\": rpc error: code = NotFound desc = could not find container \"57bca33113c4be5ed0b3bd6c01746596d8aeb741b374a30e1e3dfde44248202e\": container with ID starting with 57bca33113c4be5ed0b3bd6c01746596d8aeb741b374a30e1e3dfde44248202e not found: ID does not exist" Jul 14 19:23:23.369215 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.369160 2576 scope.go:117] "RemoveContainer" containerID="ec2e6eac606d1d5d83df338d19db38e90a71e12df54365314795dd247709c231" Jul 14 19:23:23.378499 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.378476 2576 scope.go:117] "RemoveContainer" containerID="ec2e6eac606d1d5d83df338d19db38e90a71e12df54365314795dd247709c231" Jul 14 19:23:23.378884 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:23.378861 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec2e6eac606d1d5d83df338d19db38e90a71e12df54365314795dd247709c231\": container with ID starting with ec2e6eac606d1d5d83df338d19db38e90a71e12df54365314795dd247709c231 not found: ID does not exist" containerID="ec2e6eac606d1d5d83df338d19db38e90a71e12df54365314795dd247709c231" Jul 14 19:23:23.378994 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.378890 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec2e6eac606d1d5d83df338d19db38e90a71e12df54365314795dd247709c231"} err="failed to get container status \"ec2e6eac606d1d5d83df338d19db38e90a71e12df54365314795dd247709c231\": rpc error: code = NotFound desc = could not find container \"ec2e6eac606d1d5d83df338d19db38e90a71e12df54365314795dd247709c231\": container with ID starting with ec2e6eac606d1d5d83df338d19db38e90a71e12df54365314795dd247709c231 not found: ID does not exist" Jul 14 19:23:23.381063 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.381020 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m55b7" podStartSLOduration=14.381006999 podStartE2EDuration="14.381006999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:23:23.379817439 +0000 UTC m=+756.493067309" watchObservedRunningTime="2026-07-14 19:23:23.381006999 +0000 UTC m=+756.494256847" Jul 14 19:23:23.382342 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:23.382313 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xrbqr\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" Jul 14 19:23:23.384845 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:23.384565 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jnkhg\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="f7fef8bf-ddc5-4a18-a8fe-8287de561095" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:23:23.401626 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.401555 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xjwwz" podStartSLOduration=1.148015374 podStartE2EDuration="25.401537027s" totalImagesPullingTime="24.253521653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:23:23.400379031 +0000 UTC m=+756.513628902" watchObservedRunningTime="2026-07-14 19:23:23.401537027 +0000 UTC m=+756.514786896" Jul 14 19:23:23.402570 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:23.402546 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xrbqr\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xrbqr" Jul 14 19:23:23.404890 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:23.404868 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jnkhg\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="f7fef8bf-ddc5-4a18-a8fe-8287de561095" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jnkhg" Jul 14 19:23:23.429699 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.429653 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh" podStartSLOduration=15.429636196 podStartE2EDuration="15.429636196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:23:23.428028491 +0000 UTC m=+756.541278366" watchObservedRunningTime="2026-07-14 19:23:23.429636196 +0000 UTC m=+756.542886066" Jul 14 19:23:23.514493 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.514463 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6" path="/var/lib/kubelet/pods/9eefafbf-c9b8-45fd-a5f2-8ce1239c3ad6/volumes" Jul 14 19:23:23.514856 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:23.514840 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7fef8bf-ddc5-4a18-a8fe-8287de561095" path="/var/lib/kubelet/pods/f7fef8bf-ddc5-4a18-a8fe-8287de561095/volumes" Jul 14 19:23:25.411309 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:25.411250 2576 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-5n47t_kuadrant-system_6e344d1a-88ef-45fb-8a0a-2e2b4a36513f_0(9047cbdd9461086983cf7a9c9900aacce2228bb7786778cd5aa8ce71272d2266): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-5n47t to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 14 19:23:25.411695 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:25.411332 2576 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-5n47t_kuadrant-system_6e344d1a-88ef-45fb-8a0a-2e2b4a36513f_0(9047cbdd9461086983cf7a9c9900aacce2228bb7786778cd5aa8ce71272d2266): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-5n47t to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t" Jul 14 19:23:26.382225 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:26.382184 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t" Jul 14 19:23:26.384615 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:26.384568 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-5n47t\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="6e344d1a-88ef-45fb-8a0a-2e2b4a36513f" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t" Jul 14 19:23:26.420200 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:26.420163 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t" Jul 14 19:23:26.423152 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:26.423121 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-5n47t\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="6e344d1a-88ef-45fb-8a0a-2e2b4a36513f" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t" Jul 14 19:23:26.555367 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:26.555333 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxrkp\" (UniqueName: \"kubernetes.io/projected/6e344d1a-88ef-45fb-8a0a-2e2b4a36513f-kube-api-access-pxrkp\") pod \"6e344d1a-88ef-45fb-8a0a-2e2b4a36513f\" (UID: \"6e344d1a-88ef-45fb-8a0a-2e2b4a36513f\") " Jul 14 19:23:26.555558 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:26.555406 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e344d1a-88ef-45fb-8a0a-2e2b4a36513f-extensions-socket-volume\") pod \"6e344d1a-88ef-45fb-8a0a-2e2b4a36513f\" (UID: \"6e344d1a-88ef-45fb-8a0a-2e2b4a36513f\") " Jul 14 19:23:26.555724 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:26.555700 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e344d1a-88ef-45fb-8a0a-2e2b4a36513f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6e344d1a-88ef-45fb-8a0a-2e2b4a36513f" (UID: "6e344d1a-88ef-45fb-8a0a-2e2b4a36513f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:23:26.557421 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:26.557387 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e344d1a-88ef-45fb-8a0a-2e2b4a36513f-kube-api-access-pxrkp" (OuterVolumeSpecName: "kube-api-access-pxrkp") pod "6e344d1a-88ef-45fb-8a0a-2e2b4a36513f" (UID: "6e344d1a-88ef-45fb-8a0a-2e2b4a36513f"). InnerVolumeSpecName "kube-api-access-pxrkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:23:26.656557 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:26.656475 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxrkp\" (UniqueName: \"kubernetes.io/projected/6e344d1a-88ef-45fb-8a0a-2e2b4a36513f-kube-api-access-pxrkp\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:23:26.656557 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:26.656505 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e344d1a-88ef-45fb-8a0a-2e2b4a36513f-extensions-socket-volume\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:23:27.386861 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:27.386830 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t" Jul 14 19:23:27.389445 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:27.389416 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-5n47t\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="6e344d1a-88ef-45fb-8a0a-2e2b4a36513f" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t" Jul 14 19:23:27.394872 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:27.394846 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-5n47t\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="6e344d1a-88ef-45fb-8a0a-2e2b4a36513f" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t" Jul 14 19:23:27.514041 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:27.514007 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-5n47t\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="6e344d1a-88ef-45fb-8a0a-2e2b4a36513f" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5n47t" Jul 14 19:23:27.514935 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:27.514911 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e344d1a-88ef-45fb-8a0a-2e2b4a36513f" path="/var/lib/kubelet/pods/6e344d1a-88ef-45fb-8a0a-2e2b4a36513f/volumes" Jul 14 19:23:34.374649 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:34.374619 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m55b7" Jul 14 19:23:34.375177 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:34.374698 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh" Jul 14 19:23:34.649409 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:34.649332 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp"] Jul 14 19:23:34.853675 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:34.853640 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp"] Jul 14 19:23:34.853840 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:34.853776 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp" Jul 14 19:23:34.931556 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:34.931468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrvgp\" (UniqueName: \"kubernetes.io/projected/193c76db-07b2-4d68-9136-0353baf7adff-kube-api-access-rrvgp\") pod \"kuadrant-operator-controller-manager-86878c769f-bz9vp\" (UID: \"193c76db-07b2-4d68-9136-0353baf7adff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp" Jul 14 19:23:34.931556 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:34.931518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/193c76db-07b2-4d68-9136-0353baf7adff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bz9vp\" (UID: \"193c76db-07b2-4d68-9136-0353baf7adff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp" Jul 14 19:23:35.032825 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:35.032787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrvgp\" (UniqueName: \"kubernetes.io/projected/193c76db-07b2-4d68-9136-0353baf7adff-kube-api-access-rrvgp\") pod \"kuadrant-operator-controller-manager-86878c769f-bz9vp\" (UID: \"193c76db-07b2-4d68-9136-0353baf7adff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp" Jul 14 19:23:35.032993 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:35.032845 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/193c76db-07b2-4d68-9136-0353baf7adff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bz9vp\" (UID: \"193c76db-07b2-4d68-9136-0353baf7adff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp" Jul 14 19:23:35.033240 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:35.033218 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/193c76db-07b2-4d68-9136-0353baf7adff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bz9vp\" (UID: \"193c76db-07b2-4d68-9136-0353baf7adff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp" Jul 14 19:23:35.047879 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:35.047841 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrvgp\" (UniqueName: \"kubernetes.io/projected/193c76db-07b2-4d68-9136-0353baf7adff-kube-api-access-rrvgp\") pod \"kuadrant-operator-controller-manager-86878c769f-bz9vp\" (UID: \"193c76db-07b2-4d68-9136-0353baf7adff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp" Jul 14 19:23:35.164304 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:35.164263 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp" Jul 14 19:23:35.500293 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:35.500267 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp"] Jul 14 19:23:35.503033 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:23:35.503005 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193c76db_07b2_4d68_9136_0353baf7adff.slice/crio-ad4bc39448f8ca4ef738206d7bba2009b75832e0d9d3bb59987300e50ae3c713 WatchSource:0}: Error finding container ad4bc39448f8ca4ef738206d7bba2009b75832e0d9d3bb59987300e50ae3c713: Status 404 returned error can't find the container with id ad4bc39448f8ca4ef738206d7bba2009b75832e0d9d3bb59987300e50ae3c713 Jul 14 19:23:36.436859 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:36.436827 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp" event={"ID":"193c76db-07b2-4d68-9136-0353baf7adff","Type":"ContainerStarted","Data":"7a298f13dc3bd408b388893cb4f1970495a9a026c8e50d0b1715324acca4a274"} Jul 14 19:23:36.437049 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:36.436879 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp" event={"ID":"193c76db-07b2-4d68-9136-0353baf7adff","Type":"ContainerStarted","Data":"ad4bc39448f8ca4ef738206d7bba2009b75832e0d9d3bb59987300e50ae3c713"} Jul 14 19:23:36.437049 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:36.436932 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp" Jul 14 19:23:36.458459 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:36.458403 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp" podStartSLOduration=2.458388427 podStartE2EDuration="2.458388427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:23:36.457456029 +0000 UTC m=+769.570705899" watchObservedRunningTime="2026-07-14 19:23:36.458388427 +0000 UTC m=+769.571638297" Jul 14 19:23:47.444470 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:47.444441 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp" Jul 14 19:23:47.499057 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:47.499022 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh"] Jul 14 19:23:47.499349 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:47.499307 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh" podUID="bc228e4b-7949-4456-b0bc-079aeb134472" containerName="manager" containerID="cri-o://4e6dbc018b5220af57f642f1dda1e98565c16c4ddf914e2bc10c7a942ce9e56c" gracePeriod=10 Jul 14 19:23:47.762052 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:47.762028 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh" Jul 14 19:23:47.852530 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:47.852490 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qcrm\" (UniqueName: \"kubernetes.io/projected/bc228e4b-7949-4456-b0bc-079aeb134472-kube-api-access-8qcrm\") pod \"bc228e4b-7949-4456-b0bc-079aeb134472\" (UID: \"bc228e4b-7949-4456-b0bc-079aeb134472\") " Jul 14 19:23:47.852750 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:47.852544 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc228e4b-7949-4456-b0bc-079aeb134472-extensions-socket-volume\") pod \"bc228e4b-7949-4456-b0bc-079aeb134472\" (UID: \"bc228e4b-7949-4456-b0bc-079aeb134472\") " Jul 14 19:23:47.852980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:47.852953 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc228e4b-7949-4456-b0bc-079aeb134472-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bc228e4b-7949-4456-b0bc-079aeb134472" (UID: "bc228e4b-7949-4456-b0bc-079aeb134472"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:23:47.854742 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:47.854717 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc228e4b-7949-4456-b0bc-079aeb134472-kube-api-access-8qcrm" (OuterVolumeSpecName: "kube-api-access-8qcrm") pod "bc228e4b-7949-4456-b0bc-079aeb134472" (UID: "bc228e4b-7949-4456-b0bc-079aeb134472"). InnerVolumeSpecName "kube-api-access-8qcrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:23:47.953185 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:47.953094 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qcrm\" (UniqueName: \"kubernetes.io/projected/bc228e4b-7949-4456-b0bc-079aeb134472-kube-api-access-8qcrm\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:23:47.953185 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:47.953126 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc228e4b-7949-4456-b0bc-079aeb134472-extensions-socket-volume\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:23:48.493719 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:48.493683 2576 generic.go:356] "Generic (PLEG): container finished" podID="bc228e4b-7949-4456-b0bc-079aeb134472" containerID="4e6dbc018b5220af57f642f1dda1e98565c16c4ddf914e2bc10c7a942ce9e56c" exitCode=0 Jul 14 19:23:48.494107 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:48.493733 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh" Jul 14 19:23:48.494107 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:48.493772 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh" event={"ID":"bc228e4b-7949-4456-b0bc-079aeb134472","Type":"ContainerDied","Data":"4e6dbc018b5220af57f642f1dda1e98565c16c4ddf914e2bc10c7a942ce9e56c"} Jul 14 19:23:48.494107 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:48.493832 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh" event={"ID":"bc228e4b-7949-4456-b0bc-079aeb134472","Type":"ContainerDied","Data":"f347fc6140385071b80e5264035a1a0bd8660f0b9e15015964cb1f71a0437161"} Jul 14 19:23:48.494107 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:48.493860 2576 scope.go:117] "RemoveContainer" containerID="4e6dbc018b5220af57f642f1dda1e98565c16c4ddf914e2bc10c7a942ce9e56c" Jul 14 19:23:48.502814 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:48.502794 2576 scope.go:117] "RemoveContainer" containerID="4e6dbc018b5220af57f642f1dda1e98565c16c4ddf914e2bc10c7a942ce9e56c" Jul 14 19:23:48.503151 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:23:48.503131 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e6dbc018b5220af57f642f1dda1e98565c16c4ddf914e2bc10c7a942ce9e56c\": container with ID starting with 4e6dbc018b5220af57f642f1dda1e98565c16c4ddf914e2bc10c7a942ce9e56c not found: ID does not exist" containerID="4e6dbc018b5220af57f642f1dda1e98565c16c4ddf914e2bc10c7a942ce9e56c" Jul 14 19:23:48.503214 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:48.503161 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6dbc018b5220af57f642f1dda1e98565c16c4ddf914e2bc10c7a942ce9e56c"} err="failed to get container status \"4e6dbc018b5220af57f642f1dda1e98565c16c4ddf914e2bc10c7a942ce9e56c\": rpc error: code = NotFound desc = could not find container \"4e6dbc018b5220af57f642f1dda1e98565c16c4ddf914e2bc10c7a942ce9e56c\": container with ID starting with 4e6dbc018b5220af57f642f1dda1e98565c16c4ddf914e2bc10c7a942ce9e56c not found: ID does not exist" Jul 14 19:23:48.518482 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:48.518452 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh"] Jul 14 19:23:48.522667 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:48.522641 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7gvnh"] Jul 14 19:23:49.514544 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:49.514510 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc228e4b-7949-4456-b0bc-079aeb134472" path="/var/lib/kubelet/pods/bc228e4b-7949-4456-b0bc-079aeb134472/volumes" Jul 14 19:23:50.869196 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:50.869165 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm"] Jul 14 19:23:50.869542 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:50.869497 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc228e4b-7949-4456-b0bc-079aeb134472" containerName="manager" Jul 14 19:23:50.869542 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:50.869507 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc228e4b-7949-4456-b0bc-079aeb134472" containerName="manager" Jul 14 19:23:50.869631 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:50.869572 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc228e4b-7949-4456-b0bc-079aeb134472" containerName="manager" Jul 14 19:23:50.877955 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:50.877937 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:50.882557 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:50.882537 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 14 19:23:50.882660 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:50.882537 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ntkgm\"" Jul 14 19:23:50.890244 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:50.890224 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm"] Jul 14 19:23:50.976224 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:50.976190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:50.976224 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:50.976229 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:50.976724 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:50.976263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lz6l\" (UniqueName: \"kubernetes.io/projected/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-kube-api-access-7lz6l\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:50.976724 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:50.976295 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:50.976724 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:50.976316 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:50.976724 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:50.976338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:50.976724 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:50.976365 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:50.976724 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:50.976392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:50.976724 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:50.976417 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.077271 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.077227 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.077437 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.077289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.077437 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.077327 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.077437 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.077361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.077437 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.077419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.077647 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.077473 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lz6l\" (UniqueName: \"kubernetes.io/projected/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-kube-api-access-7lz6l\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.077647 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.077507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.077759 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.077665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.077759 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.077715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.077867 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.077769 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.078047 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.078018 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.078162 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.078102 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.078162 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.078107 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.078330 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.078309 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.079967 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.079946 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.080058 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.079968 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.086840 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.086800 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lz6l\" (UniqueName: \"kubernetes.io/projected/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-kube-api-access-7lz6l\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.086972 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.086955 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3ab25f-fc0d-4d53-97c7-48048ff7a532-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-fcpcm\" (UID: \"3e3ab25f-fc0d-4d53-97c7-48048ff7a532\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.188420 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.188345 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:23:51.339122 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.339096 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm"] Jul 14 19:23:51.341929 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:23:51.341900 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3ab25f_fc0d_4d53_97c7_48048ff7a532.slice/crio-aacf89470f765e6604ada653c83f21dedd3b884f5d2f1989f5edf91de42d6ecf WatchSource:0}: Error finding container aacf89470f765e6604ada653c83f21dedd3b884f5d2f1989f5edf91de42d6ecf: Status 404 returned error can't find the container with id aacf89470f765e6604ada653c83f21dedd3b884f5d2f1989f5edf91de42d6ecf Jul 14 19:23:51.513981 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:23:51.513896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" event={"ID":"3e3ab25f-fc0d-4d53-97c7-48048ff7a532","Type":"ContainerStarted","Data":"aacf89470f765e6604ada653c83f21dedd3b884f5d2f1989f5edf91de42d6ecf"} Jul 14 19:24:16.032523 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:16.032489 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:24:16.032827 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:16.032560 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:24:16.032827 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:16.032604 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:24:16.625982 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:16.625944 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" event={"ID":"3e3ab25f-fc0d-4d53-97c7-48048ff7a532","Type":"ContainerStarted","Data":"36b7137ab8069d5ee285fc8d0e943c3bb04a69b7876f56c6f5861f2315ce1867"} Jul 14 19:24:16.647999 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:16.647947 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" podStartSLOduration=1.959894361 podStartE2EDuration="26.647930862s" totalImagesPullingTime="24.688036501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:24:16.646659965 +0000 UTC m=+809.759909853" watchObservedRunningTime="2026-07-14 19:24:16.647930862 +0000 UTC m=+809.761180732" Jul 14 19:24:17.189040 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:17.189001 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:24:17.193937 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:17.193908 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:24:17.631219 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:17.631188 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:24:17.632393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:17.632373 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:24:18.718330 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:18.718299 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-qck26"] Jul 14 19:24:18.722023 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:18.722006 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-qck26" Jul 14 19:24:18.724684 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:18.724667 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qdr5n\"" Jul 14 19:24:18.728654 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:18.728633 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qck26"] Jul 14 19:24:18.834313 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:18.834265 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk88t\" (UniqueName: \"kubernetes.io/projected/fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1-kube-api-access-dk88t\") pod \"authorino-75d556bd77-qck26\" (UID: \"fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1\") " pod="kuadrant-system/authorino-75d556bd77-qck26" Jul 14 19:24:18.935137 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:18.935091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk88t\" (UniqueName: \"kubernetes.io/projected/fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1-kube-api-access-dk88t\") pod \"authorino-75d556bd77-qck26\" (UID: \"fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1\") " pod="kuadrant-system/authorino-75d556bd77-qck26" Jul 14 19:24:18.943973 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:18.943933 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk88t\" (UniqueName: \"kubernetes.io/projected/fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1-kube-api-access-dk88t\") pod \"authorino-75d556bd77-qck26\" (UID: \"fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1\") " pod="kuadrant-system/authorino-75d556bd77-qck26" Jul 14 19:24:19.031955 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:19.031850 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-qck26" Jul 14 19:24:19.168616 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:19.168567 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qck26"] Jul 14 19:24:19.172436 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:24:19.172411 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe43ab79_09bc_4f63_b7e3_ac54cb86a3b1.slice/crio-75858e7edd3ec57ea1f4e0d1e30f68aeb692962611aaa7c71d2b94c573849e73 WatchSource:0}: Error finding container 75858e7edd3ec57ea1f4e0d1e30f68aeb692962611aaa7c71d2b94c573849e73: Status 404 returned error can't find the container with id 75858e7edd3ec57ea1f4e0d1e30f68aeb692962611aaa7c71d2b94c573849e73 Jul 14 19:24:19.641026 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:19.640989 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-qck26" event={"ID":"fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1","Type":"ContainerStarted","Data":"75858e7edd3ec57ea1f4e0d1e30f68aeb692962611aaa7c71d2b94c573849e73"} Jul 14 19:24:23.666057 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:23.666019 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-qck26" event={"ID":"fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1","Type":"ContainerStarted","Data":"cc73dbc01d291177dd443b20d0bff36f3efd493f00661365cb39ab6f48bfdb4a"} Jul 14 19:24:23.682238 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:23.682193 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-qck26" podStartSLOduration=1.595460138 podStartE2EDuration="5.682179335s" totalImagesPullingTime="4.086719197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:24:23.681338629 +0000 UTC m=+816.794588513" watchObservedRunningTime="2026-07-14 19:24:23.682179335 +0000 UTC m=+816.795429204" Jul 14 19:24:39.402979 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:39.402940 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nfzrx"] Jul 14 19:24:39.406839 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:39.406813 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nfzrx" Jul 14 19:24:39.409616 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:39.409593 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 14 19:24:39.409757 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:39.409608 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 14 19:24:39.409757 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:39.409611 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 19:24:39.410717 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:39.410701 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-p6249\"" Jul 14 19:24:39.415690 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:39.415666 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nfzrx"] Jul 14 19:24:39.418016 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:39.417990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/33693bfd-f953-4e4d-b668-b478d36ad79c-data\") pod \"postgres-69fb4594cf-nfzrx\" (UID: \"33693bfd-f953-4e4d-b668-b478d36ad79c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nfzrx" Jul 14 19:24:39.418143 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:39.418079 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjf57\" (UniqueName: \"kubernetes.io/projected/33693bfd-f953-4e4d-b668-b478d36ad79c-kube-api-access-mjf57\") pod \"postgres-69fb4594cf-nfzrx\" (UID: \"33693bfd-f953-4e4d-b668-b478d36ad79c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nfzrx" Jul 14 19:24:39.518724 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:39.518697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjf57\" (UniqueName: \"kubernetes.io/projected/33693bfd-f953-4e4d-b668-b478d36ad79c-kube-api-access-mjf57\") pod \"postgres-69fb4594cf-nfzrx\" (UID: \"33693bfd-f953-4e4d-b668-b478d36ad79c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nfzrx" Jul 14 19:24:39.518891 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:39.518771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/33693bfd-f953-4e4d-b668-b478d36ad79c-data\") pod \"postgres-69fb4594cf-nfzrx\" (UID: \"33693bfd-f953-4e4d-b668-b478d36ad79c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nfzrx" Jul 14 19:24:39.519111 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:39.519087 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/33693bfd-f953-4e4d-b668-b478d36ad79c-data\") pod \"postgres-69fb4594cf-nfzrx\" (UID: \"33693bfd-f953-4e4d-b668-b478d36ad79c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nfzrx" Jul 14 19:24:39.526951 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:39.526924 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjf57\" (UniqueName: \"kubernetes.io/projected/33693bfd-f953-4e4d-b668-b478d36ad79c-kube-api-access-mjf57\") pod \"postgres-69fb4594cf-nfzrx\" (UID: \"33693bfd-f953-4e4d-b668-b478d36ad79c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nfzrx" Jul 14 19:24:39.718702 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:39.718616 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nfzrx" Jul 14 19:24:39.851593 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:39.851555 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nfzrx"] Jul 14 19:24:39.853959 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:24:39.853931 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33693bfd_f953_4e4d_b668_b478d36ad79c.slice/crio-a10edf198853750d1bf11d13e8953878f97aa6b7ad1599a49fb8a78d69046b24 WatchSource:0}: Error finding container a10edf198853750d1bf11d13e8953878f97aa6b7ad1599a49fb8a78d69046b24: Status 404 returned error can't find the container with id a10edf198853750d1bf11d13e8953878f97aa6b7ad1599a49fb8a78d69046b24 Jul 14 19:24:40.745751 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:40.745679 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nfzrx" event={"ID":"33693bfd-f953-4e4d-b668-b478d36ad79c","Type":"ContainerStarted","Data":"a10edf198853750d1bf11d13e8953878f97aa6b7ad1599a49fb8a78d69046b24"} Jul 14 19:24:44.769846 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:44.769811 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nfzrx" event={"ID":"33693bfd-f953-4e4d-b668-b478d36ad79c","Type":"ContainerStarted","Data":"dbc0df8c306a0969c944233b260ebfa21ac695cd8dc207d17c3db2477c3b6e91"} Jul 14 19:24:44.770198 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:44.770004 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nfzrx" Jul 14 19:24:44.785812 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:44.785763 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nfzrx" podStartSLOduration=0.938228702 podStartE2EDuration="5.78574695s" totalImagesPullingTime="4.847518248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:24:44.784649664 +0000 UTC m=+837.897899534" watchObservedRunningTime="2026-07-14 19:24:44.78574695 +0000 UTC m=+837.898996819" Jul 14 19:24:50.803855 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:50.803826 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nfzrx" Jul 14 19:24:53.133557 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:53.133511 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qck26"] Jul 14 19:24:53.133991 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:53.133774 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-qck26" podUID="fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1" containerName="authorino" containerID="cri-o://cc73dbc01d291177dd443b20d0bff36f3efd493f00661365cb39ab6f48bfdb4a" gracePeriod=30 Jul 14 19:24:53.374253 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:53.374225 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-qck26" Jul 14 19:24:53.452915 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:53.452828 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk88t\" (UniqueName: \"kubernetes.io/projected/fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1-kube-api-access-dk88t\") pod \"fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1\" (UID: \"fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1\") " Jul 14 19:24:53.454976 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:53.454949 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1-kube-api-access-dk88t" (OuterVolumeSpecName: "kube-api-access-dk88t") pod "fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1" (UID: "fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1"). InnerVolumeSpecName "kube-api-access-dk88t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:24:53.554141 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:53.554107 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk88t\" (UniqueName: \"kubernetes.io/projected/fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1-kube-api-access-dk88t\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:24:53.811234 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:53.811140 2576 generic.go:356] "Generic (PLEG): container finished" podID="fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1" containerID="cc73dbc01d291177dd443b20d0bff36f3efd493f00661365cb39ab6f48bfdb4a" exitCode=0 Jul 14 19:24:53.811234 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:53.811194 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-qck26" Jul 14 19:24:53.811477 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:53.811227 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-qck26" event={"ID":"fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1","Type":"ContainerDied","Data":"cc73dbc01d291177dd443b20d0bff36f3efd493f00661365cb39ab6f48bfdb4a"} Jul 14 19:24:53.811477 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:53.811285 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-qck26" event={"ID":"fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1","Type":"ContainerDied","Data":"75858e7edd3ec57ea1f4e0d1e30f68aeb692962611aaa7c71d2b94c573849e73"} Jul 14 19:24:53.811477 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:53.811309 2576 scope.go:117] "RemoveContainer" containerID="cc73dbc01d291177dd443b20d0bff36f3efd493f00661365cb39ab6f48bfdb4a" Jul 14 19:24:53.825801 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:53.825779 2576 scope.go:117] "RemoveContainer" containerID="cc73dbc01d291177dd443b20d0bff36f3efd493f00661365cb39ab6f48bfdb4a" Jul 14 19:24:53.826128 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:24:53.826106 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc73dbc01d291177dd443b20d0bff36f3efd493f00661365cb39ab6f48bfdb4a\": container with ID starting with cc73dbc01d291177dd443b20d0bff36f3efd493f00661365cb39ab6f48bfdb4a not found: ID does not exist" containerID="cc73dbc01d291177dd443b20d0bff36f3efd493f00661365cb39ab6f48bfdb4a" Jul 14 19:24:53.826173 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:53.826138 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc73dbc01d291177dd443b20d0bff36f3efd493f00661365cb39ab6f48bfdb4a"} err="failed to get container status \"cc73dbc01d291177dd443b20d0bff36f3efd493f00661365cb39ab6f48bfdb4a\": rpc error: code = NotFound desc = could not find container \"cc73dbc01d291177dd443b20d0bff36f3efd493f00661365cb39ab6f48bfdb4a\": container with ID starting with cc73dbc01d291177dd443b20d0bff36f3efd493f00661365cb39ab6f48bfdb4a not found: ID does not exist" Jul 14 19:24:53.832636 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:53.832604 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qck26"] Jul 14 19:24:53.841216 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:53.841186 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qck26"] Jul 14 19:24:55.515389 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:24:55.515359 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1" path="/var/lib/kubelet/pods/fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1/volumes" Jul 14 19:25:41.034461 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:41.034426 2576 generic.go:356] "Generic (PLEG): container finished" podID="3e3ab25f-fc0d-4d53-97c7-48048ff7a532" containerID="36b7137ab8069d5ee285fc8d0e943c3bb04a69b7876f56c6f5861f2315ce1867" exitCode=0 Jul 14 19:25:41.034929 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:41.034507 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" event={"ID":"3e3ab25f-fc0d-4d53-97c7-48048ff7a532","Type":"ContainerDied","Data":"36b7137ab8069d5ee285fc8d0e943c3bb04a69b7876f56c6f5861f2315ce1867"} Jul 14 19:25:41.034929 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:41.034894 2576 scope.go:117] "RemoveContainer" containerID="36b7137ab8069d5ee285fc8d0e943c3bb04a69b7876f56c6f5861f2315ce1867" Jul 14 19:25:41.036009 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:41.035974 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:25:41.036078 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:41.036055 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:25:41.036116 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:41.036089 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:25:42.041208 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:42.041180 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" event={"ID":"3e3ab25f-fc0d-4d53-97c7-48048ff7a532","Type":"ContainerStarted","Data":"320dff18d9d0a80dd719faca0347333bfc305f282731b83a6d39d782f15c7851"} Jul 14 19:25:42.189335 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:42.189304 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:25:42.189935 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:42.189358 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-fcpcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 14 19:25:42.189935 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:42.189690 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" podUID="3e3ab25f-fc0d-4d53-97c7-48048ff7a532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 14 19:25:43.188941 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:43.188899 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-fcpcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 14 19:25:43.189441 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:43.188963 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" podUID="3e3ab25f-fc0d-4d53-97c7-48048ff7a532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 14 19:25:44.189006 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:44.188964 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-fcpcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 14 19:25:44.189562 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:44.189037 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" podUID="3e3ab25f-fc0d-4d53-97c7-48048ff7a532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 14 19:25:45.189326 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:45.189284 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-fcpcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 14 19:25:45.189830 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:45.189352 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" podUID="3e3ab25f-fc0d-4d53-97c7-48048ff7a532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 14 19:25:46.192957 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:46.192925 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:25:46.193379 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:46.193216 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:25:46.193975 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:46.193958 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fcpcm" Jul 14 19:25:47.435486 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:47.435460 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:25:47.435941 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:47.435518 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:25:58.608372 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.608331 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l"] Jul 14 19:25:58.608872 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.608853 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1" containerName="authorino" Jul 14 19:25:58.608872 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.608873 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1" containerName="authorino" Jul 14 19:25:58.608987 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.608942 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe43ab79-09bc-4f63-b7e3-ac54cb86a3b1" containerName="authorino" Jul 14 19:25:58.612299 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.612277 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.614843 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.614825 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 14 19:25:58.616043 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.616026 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 14 19:25:58.616129 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.616114 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cxkgd\"" Jul 14 19:25:58.616192 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.616169 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 14 19:25:58.621152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.621126 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l"] Jul 14 19:25:58.673805 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.673767 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3593aa0-22b9-42de-a13f-1ca28558e244-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vt94l\" (UID: \"b3593aa0-22b9-42de-a13f-1ca28558e244\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.673805 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.673811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3593aa0-22b9-42de-a13f-1ca28558e244-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vt94l\" (UID: \"b3593aa0-22b9-42de-a13f-1ca28558e244\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.674090 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.673828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3593aa0-22b9-42de-a13f-1ca28558e244-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vt94l\" (UID: \"b3593aa0-22b9-42de-a13f-1ca28558e244\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.674090 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.673843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3593aa0-22b9-42de-a13f-1ca28558e244-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vt94l\" (UID: \"b3593aa0-22b9-42de-a13f-1ca28558e244\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.674090 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.673867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wv8f\" (UniqueName: \"kubernetes.io/projected/b3593aa0-22b9-42de-a13f-1ca28558e244-kube-api-access-4wv8f\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vt94l\" (UID: \"b3593aa0-22b9-42de-a13f-1ca28558e244\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.674090 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.673984 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3593aa0-22b9-42de-a13f-1ca28558e244-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vt94l\" (UID: \"b3593aa0-22b9-42de-a13f-1ca28558e244\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.775441 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.775401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3593aa0-22b9-42de-a13f-1ca28558e244-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vt94l\" (UID: \"b3593aa0-22b9-42de-a13f-1ca28558e244\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.775441 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.775445 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3593aa0-22b9-42de-a13f-1ca28558e244-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vt94l\" (UID: \"b3593aa0-22b9-42de-a13f-1ca28558e244\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.775723 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.775463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3593aa0-22b9-42de-a13f-1ca28558e244-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vt94l\" (UID: \"b3593aa0-22b9-42de-a13f-1ca28558e244\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.775723 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.775480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3593aa0-22b9-42de-a13f-1ca28558e244-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vt94l\" (UID: \"b3593aa0-22b9-42de-a13f-1ca28558e244\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.775723 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.775506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wv8f\" (UniqueName: \"kubernetes.io/projected/b3593aa0-22b9-42de-a13f-1ca28558e244-kube-api-access-4wv8f\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vt94l\" (UID: \"b3593aa0-22b9-42de-a13f-1ca28558e244\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.775723 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.775568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3593aa0-22b9-42de-a13f-1ca28558e244-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vt94l\" (UID: \"b3593aa0-22b9-42de-a13f-1ca28558e244\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.775977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.775948 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3593aa0-22b9-42de-a13f-1ca28558e244-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vt94l\" (UID: \"b3593aa0-22b9-42de-a13f-1ca28558e244\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.776045 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.775982 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3593aa0-22b9-42de-a13f-1ca28558e244-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vt94l\" (UID: \"b3593aa0-22b9-42de-a13f-1ca28558e244\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.776045 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.776026 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3593aa0-22b9-42de-a13f-1ca28558e244-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vt94l\" (UID: \"b3593aa0-22b9-42de-a13f-1ca28558e244\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.778298 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.778270 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3593aa0-22b9-42de-a13f-1ca28558e244-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vt94l\" (UID: \"b3593aa0-22b9-42de-a13f-1ca28558e244\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.778433 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.778360 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3593aa0-22b9-42de-a13f-1ca28558e244-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vt94l\" (UID: \"b3593aa0-22b9-42de-a13f-1ca28558e244\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.784977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.784948 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wv8f\" (UniqueName: \"kubernetes.io/projected/b3593aa0-22b9-42de-a13f-1ca28558e244-kube-api-access-4wv8f\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vt94l\" (UID: \"b3593aa0-22b9-42de-a13f-1ca28558e244\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:58.923513 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:58.923424 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:25:59.062822 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:59.062798 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l"] Jul 14 19:25:59.065205 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:25:59.065165 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3593aa0_22b9_42de_a13f_1ca28558e244.slice/crio-75b83ecd0cf3a9748d595a1a01d1d3114fb080a6776284b1fd9543c6ac00b08a WatchSource:0}: Error finding container 75b83ecd0cf3a9748d595a1a01d1d3114fb080a6776284b1fd9543c6ac00b08a: Status 404 returned error can't find the container with id 75b83ecd0cf3a9748d595a1a01d1d3114fb080a6776284b1fd9543c6ac00b08a Jul 14 19:25:59.127727 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:25:59.127690 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" event={"ID":"b3593aa0-22b9-42de-a13f-1ca28558e244","Type":"ContainerStarted","Data":"75b83ecd0cf3a9748d595a1a01d1d3114fb080a6776284b1fd9543c6ac00b08a"} Jul 14 19:26:01.139146 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:01.139099 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" event={"ID":"b3593aa0-22b9-42de-a13f-1ca28558e244","Type":"ContainerStarted","Data":"3c0bc3088eb7de23d152cc103f7df3c2e38e99671ee4d476e24fa0739ef98dd8"} Jul 14 19:26:01.164751 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:01.159323 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" podStartSLOduration=2.120474026 podStartE2EDuration="3.159301727s" totalImagesPullingTime="1.038827701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:26:01.156858845 +0000 UTC m=+914.270108739" watchObservedRunningTime="2026-07-14 19:26:01.159301727 +0000 UTC m=+914.272551598" Jul 14 19:26:08.924363 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:08.924320 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:26:08.925055 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:08.924433 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:26:08.937112 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:08.937082 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:26:09.187523 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:09.187438 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vt94l" Jul 14 19:26:19.295142 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.295095 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s"] Jul 14 19:26:19.297835 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.297810 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.300656 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.300559 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 14 19:26:19.313241 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.313215 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s"] Jul 14 19:26:19.464495 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.464460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5694bb63-fad1-466f-8295-c041cef8b0c3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s\" (UID: \"5694bb63-fad1-466f-8295-c041cef8b0c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.464495 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.464497 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9t78\" (UniqueName: \"kubernetes.io/projected/5694bb63-fad1-466f-8295-c041cef8b0c3-kube-api-access-h9t78\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s\" (UID: \"5694bb63-fad1-466f-8295-c041cef8b0c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.464750 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.464544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5694bb63-fad1-466f-8295-c041cef8b0c3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s\" (UID: \"5694bb63-fad1-466f-8295-c041cef8b0c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.464750 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.464566 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5694bb63-fad1-466f-8295-c041cef8b0c3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s\" (UID: \"5694bb63-fad1-466f-8295-c041cef8b0c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.464750 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.464673 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5694bb63-fad1-466f-8295-c041cef8b0c3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s\" (UID: \"5694bb63-fad1-466f-8295-c041cef8b0c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.464750 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.464729 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5694bb63-fad1-466f-8295-c041cef8b0c3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s\" (UID: \"5694bb63-fad1-466f-8295-c041cef8b0c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.566119 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.566032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5694bb63-fad1-466f-8295-c041cef8b0c3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s\" (UID: \"5694bb63-fad1-466f-8295-c041cef8b0c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.566119 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.566075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9t78\" (UniqueName: \"kubernetes.io/projected/5694bb63-fad1-466f-8295-c041cef8b0c3-kube-api-access-h9t78\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s\" (UID: \"5694bb63-fad1-466f-8295-c041cef8b0c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.566119 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.566122 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5694bb63-fad1-466f-8295-c041cef8b0c3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s\" (UID: \"5694bb63-fad1-466f-8295-c041cef8b0c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.566395 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.566138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5694bb63-fad1-466f-8295-c041cef8b0c3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s\" (UID: \"5694bb63-fad1-466f-8295-c041cef8b0c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.566395 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.566165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5694bb63-fad1-466f-8295-c041cef8b0c3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s\" (UID: \"5694bb63-fad1-466f-8295-c041cef8b0c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.566395 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.566192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5694bb63-fad1-466f-8295-c041cef8b0c3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s\" (UID: \"5694bb63-fad1-466f-8295-c041cef8b0c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.566563 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.566536 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5694bb63-fad1-466f-8295-c041cef8b0c3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s\" (UID: \"5694bb63-fad1-466f-8295-c041cef8b0c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.566678 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.566612 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5694bb63-fad1-466f-8295-c041cef8b0c3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s\" (UID: \"5694bb63-fad1-466f-8295-c041cef8b0c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.566731 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.566695 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5694bb63-fad1-466f-8295-c041cef8b0c3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s\" (UID: \"5694bb63-fad1-466f-8295-c041cef8b0c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.568488 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.568467 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5694bb63-fad1-466f-8295-c041cef8b0c3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s\" (UID: \"5694bb63-fad1-466f-8295-c041cef8b0c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.568688 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.568672 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5694bb63-fad1-466f-8295-c041cef8b0c3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s\" (UID: \"5694bb63-fad1-466f-8295-c041cef8b0c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.575280 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.575260 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9t78\" (UniqueName: \"kubernetes.io/projected/5694bb63-fad1-466f-8295-c041cef8b0c3-kube-api-access-h9t78\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s\" (UID: \"5694bb63-fad1-466f-8295-c041cef8b0c3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.608679 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.608650 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:19.751506 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:19.751482 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s"] Jul 14 19:26:19.753816 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:26:19.753788 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5694bb63_fad1_466f_8295_c041cef8b0c3.slice/crio-f10ad91490ab65c1158b99242ac57438ba80a1e96808d4f1d9804dedc201b0d4 WatchSource:0}: Error finding container f10ad91490ab65c1158b99242ac57438ba80a1e96808d4f1d9804dedc201b0d4: Status 404 returned error can't find the container with id f10ad91490ab65c1158b99242ac57438ba80a1e96808d4f1d9804dedc201b0d4 Jul 14 19:26:20.230553 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:20.230508 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" event={"ID":"5694bb63-fad1-466f-8295-c041cef8b0c3","Type":"ContainerStarted","Data":"2e7e62fe9fc0dc712dc3d7f216e085cfb37959be930a9eda28e5c98ebfbfa03e"} Jul 14 19:26:20.230753 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:20.230569 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" event={"ID":"5694bb63-fad1-466f-8295-c041cef8b0c3","Type":"ContainerStarted","Data":"f10ad91490ab65c1158b99242ac57438ba80a1e96808d4f1d9804dedc201b0d4"} Jul 14 19:26:20.247964 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:20.247855 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" podStartSLOduration=1.004889321 podStartE2EDuration="1.247835452s" totalImagesPullingTime="242.946131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:26:20.247350531 +0000 UTC m=+933.360600400" watchObservedRunningTime="2026-07-14 19:26:20.247835452 +0000 UTC m=+933.361085328" Jul 14 19:26:29.609020 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:29.608981 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:29.609491 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:29.609050 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:29.621828 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:29.621798 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:26:30.287863 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:26:30.287830 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s" Jul 14 19:30:47.463991 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:30:47.463959 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:30:47.466489 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:30:47.466470 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:33:48.242432 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:48.242395 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp"] Jul 14 19:33:48.243078 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:48.242634 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp" podUID="193c76db-07b2-4d68-9136-0353baf7adff" containerName="manager" containerID="cri-o://7a298f13dc3bd408b388893cb4f1970495a9a026c8e50d0b1715324acca4a274" gracePeriod=10 Jul 14 19:33:49.698927 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:49.698902 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp" Jul 14 19:33:49.763646 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:49.763549 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/193c76db-07b2-4d68-9136-0353baf7adff-extensions-socket-volume\") pod \"193c76db-07b2-4d68-9136-0353baf7adff\" (UID: \"193c76db-07b2-4d68-9136-0353baf7adff\") " Jul 14 19:33:49.763793 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:49.763653 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrvgp\" (UniqueName: \"kubernetes.io/projected/193c76db-07b2-4d68-9136-0353baf7adff-kube-api-access-rrvgp\") pod \"193c76db-07b2-4d68-9136-0353baf7adff\" (UID: \"193c76db-07b2-4d68-9136-0353baf7adff\") " Jul 14 19:33:49.764004 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:49.763979 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193c76db-07b2-4d68-9136-0353baf7adff-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "193c76db-07b2-4d68-9136-0353baf7adff" (UID: "193c76db-07b2-4d68-9136-0353baf7adff"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:33:49.765717 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:49.765688 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193c76db-07b2-4d68-9136-0353baf7adff-kube-api-access-rrvgp" (OuterVolumeSpecName: "kube-api-access-rrvgp") pod "193c76db-07b2-4d68-9136-0353baf7adff" (UID: "193c76db-07b2-4d68-9136-0353baf7adff"). InnerVolumeSpecName "kube-api-access-rrvgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:33:49.864521 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:49.864483 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrvgp\" (UniqueName: \"kubernetes.io/projected/193c76db-07b2-4d68-9136-0353baf7adff-kube-api-access-rrvgp\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:33:49.864521 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:49.864514 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/193c76db-07b2-4d68-9136-0353baf7adff-extensions-socket-volume\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:33:50.407700 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:50.407670 2576 generic.go:356] "Generic (PLEG): container finished" podID="193c76db-07b2-4d68-9136-0353baf7adff" containerID="7a298f13dc3bd408b388893cb4f1970495a9a026c8e50d0b1715324acca4a274" exitCode=0 Jul 14 19:33:50.407878 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:50.407712 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp" event={"ID":"193c76db-07b2-4d68-9136-0353baf7adff","Type":"ContainerDied","Data":"7a298f13dc3bd408b388893cb4f1970495a9a026c8e50d0b1715324acca4a274"} Jul 14 19:33:50.407878 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:50.407776 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp" event={"ID":"193c76db-07b2-4d68-9136-0353baf7adff","Type":"ContainerDied","Data":"ad4bc39448f8ca4ef738206d7bba2009b75832e0d9d3bb59987300e50ae3c713"} Jul 14 19:33:50.407878 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:50.407714 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp" Jul 14 19:33:50.407878 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:50.407796 2576 scope.go:117] "RemoveContainer" containerID="7a298f13dc3bd408b388893cb4f1970495a9a026c8e50d0b1715324acca4a274" Jul 14 19:33:50.417308 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:50.417290 2576 scope.go:117] "RemoveContainer" containerID="7a298f13dc3bd408b388893cb4f1970495a9a026c8e50d0b1715324acca4a274" Jul 14 19:33:50.417571 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:33:50.417550 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a298f13dc3bd408b388893cb4f1970495a9a026c8e50d0b1715324acca4a274\": container with ID starting with 7a298f13dc3bd408b388893cb4f1970495a9a026c8e50d0b1715324acca4a274 not found: ID does not exist" containerID="7a298f13dc3bd408b388893cb4f1970495a9a026c8e50d0b1715324acca4a274" Jul 14 19:33:50.417640 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:50.417613 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a298f13dc3bd408b388893cb4f1970495a9a026c8e50d0b1715324acca4a274"} err="failed to get container status \"7a298f13dc3bd408b388893cb4f1970495a9a026c8e50d0b1715324acca4a274\": rpc error: code = NotFound desc = could not find container \"7a298f13dc3bd408b388893cb4f1970495a9a026c8e50d0b1715324acca4a274\": container with ID starting with 7a298f13dc3bd408b388893cb4f1970495a9a026c8e50d0b1715324acca4a274 not found: ID does not exist" Jul 14 19:33:50.430676 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:50.430647 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp"] Jul 14 19:33:50.435219 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:50.435194 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bz9vp"] Jul 14 19:33:51.515257 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:33:51.515222 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="193c76db-07b2-4d68-9136-0353baf7adff" path="/var/lib/kubelet/pods/193c76db-07b2-4d68-9136-0353baf7adff/volumes" Jul 14 19:34:49.515338 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:49.515301 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kxn8z"] Jul 14 19:34:49.515757 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:49.515661 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="193c76db-07b2-4d68-9136-0353baf7adff" containerName="manager" Jul 14 19:34:49.515757 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:49.515672 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="193c76db-07b2-4d68-9136-0353baf7adff" containerName="manager" Jul 14 19:34:49.515757 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:49.515727 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="193c76db-07b2-4d68-9136-0353baf7adff" containerName="manager" Jul 14 19:34:49.518412 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:49.518397 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kxn8z" Jul 14 19:34:49.521224 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:49.521203 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sncpl\"" Jul 14 19:34:49.530379 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:49.530357 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kxn8z"] Jul 14 19:34:49.617043 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:49.617000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68tjw\" (UniqueName: \"kubernetes.io/projected/1c13f2e4-d0b8-4ca5-8f34-2238c7060164-kube-api-access-68tjw\") pod \"kuadrant-operator-controller-manager-86878c769f-kxn8z\" (UID: \"1c13f2e4-d0b8-4ca5-8f34-2238c7060164\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kxn8z" Jul 14 19:34:49.617236 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:49.617096 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c13f2e4-d0b8-4ca5-8f34-2238c7060164-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-kxn8z\" (UID: \"1c13f2e4-d0b8-4ca5-8f34-2238c7060164\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kxn8z" Jul 14 19:34:49.717950 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:49.717903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c13f2e4-d0b8-4ca5-8f34-2238c7060164-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-kxn8z\" (UID: \"1c13f2e4-d0b8-4ca5-8f34-2238c7060164\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kxn8z" Jul 14 19:34:49.718156 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:49.718007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68tjw\" (UniqueName: \"kubernetes.io/projected/1c13f2e4-d0b8-4ca5-8f34-2238c7060164-kube-api-access-68tjw\") pod \"kuadrant-operator-controller-manager-86878c769f-kxn8z\" (UID: \"1c13f2e4-d0b8-4ca5-8f34-2238c7060164\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kxn8z" Jul 14 19:34:49.718316 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:49.718290 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c13f2e4-d0b8-4ca5-8f34-2238c7060164-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-kxn8z\" (UID: \"1c13f2e4-d0b8-4ca5-8f34-2238c7060164\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kxn8z" Jul 14 19:34:49.729417 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:49.729392 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68tjw\" (UniqueName: \"kubernetes.io/projected/1c13f2e4-d0b8-4ca5-8f34-2238c7060164-kube-api-access-68tjw\") pod \"kuadrant-operator-controller-manager-86878c769f-kxn8z\" (UID: \"1c13f2e4-d0b8-4ca5-8f34-2238c7060164\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kxn8z" Jul 14 19:34:49.829234 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:49.829205 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kxn8z" Jul 14 19:34:49.972319 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:49.972293 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kxn8z"] Jul 14 19:34:49.974761 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:34:49.974735 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c13f2e4_d0b8_4ca5_8f34_2238c7060164.slice/crio-93782b85549c2a964e78ad1a303498e2974461a4fee4cd0c58af9d7a93252623 WatchSource:0}: Error finding container 93782b85549c2a964e78ad1a303498e2974461a4fee4cd0c58af9d7a93252623: Status 404 returned error can't find the container with id 93782b85549c2a964e78ad1a303498e2974461a4fee4cd0c58af9d7a93252623 Jul 14 19:34:50.694973 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:50.694931 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kxn8z" event={"ID":"1c13f2e4-d0b8-4ca5-8f34-2238c7060164","Type":"ContainerStarted","Data":"1c1561f4e60c2dae5c1830f089398bfd0eaea97fff112cba9d9d171854d46d6e"} Jul 14 19:34:50.695453 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:50.694993 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kxn8z" event={"ID":"1c13f2e4-d0b8-4ca5-8f34-2238c7060164","Type":"ContainerStarted","Data":"93782b85549c2a964e78ad1a303498e2974461a4fee4cd0c58af9d7a93252623"} Jul 14 19:34:50.695453 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:50.695112 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kxn8z" Jul 14 19:34:50.717924 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:34:50.717871 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kxn8z" podStartSLOduration=1.7178523270000001 podStartE2EDuration="1.717852327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:34:50.71564255 +0000 UTC m=+1443.828892421" watchObservedRunningTime="2026-07-14 19:34:50.717852327 +0000 UTC m=+1443.831102196" Jul 14 19:35:01.702839 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:35:01.702807 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kxn8z" Jul 14 19:35:47.495003 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:35:47.494888 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:35:47.499495 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:35:47.499472 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:40:47.527001 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:40:47.526894 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:40:47.533780 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:40:47.531497 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:41:20.122482 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.122449 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl"] Jul 14 19:41:20.126300 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.126145 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.128783 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.128762 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-102a2069-openshift-default-dockercfg-r46lg\"" Jul 14 19:41:20.143610 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.143563 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl"] Jul 14 19:41:20.176065 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.176030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-podinfo\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.176248 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.176072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-credential-socket\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.176248 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.176097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-workload-certs\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.176248 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.176122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-data\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.176248 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.176140 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-workload-socket\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.176248 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.176172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-envoy\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.176248 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.176236 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs6b5\" (UniqueName: \"kubernetes.io/projected/74e5ce24-18f9-4ae3-a604-ae93f57743fa-kube-api-access-zs6b5\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.176448 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.176315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-token\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.176448 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.176365 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istiod-ca-cert\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.277743 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.277702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs6b5\" (UniqueName: \"kubernetes.io/projected/74e5ce24-18f9-4ae3-a604-ae93f57743fa-kube-api-access-zs6b5\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.277743 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.277755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-token\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.278034 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.277821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istiod-ca-cert\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.278034 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.277873 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-podinfo\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.278034 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.277904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-credential-socket\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.278034 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.277919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-workload-certs\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.278034 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.277937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-data\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.278034 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.277956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-workload-socket\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.278034 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.277990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-envoy\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.278894 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.278843 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istiod-ca-cert\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.279092 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.279068 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-data\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.279205 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.279184 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-credential-socket\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.279370 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.279343 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-workload-socket\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.279468 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.279369 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-workload-certs\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.281103 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.281084 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-envoy\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.281205 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.281183 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-podinfo\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.286051 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.286030 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-token\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.286696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.286674 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs6b5\" (UniqueName: \"kubernetes.io/projected/74e5ce24-18f9-4ae3-a604-ae93f57743fa-kube-api-access-zs6b5\") pod \"e2e-ait-102a2069-openshift-default-684db9f89-425wl\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.437157 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.437057 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:20.584630 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.584605 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl"] Jul 14 19:41:20.586088 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:41:20.586059 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e5ce24_18f9_4ae3_a604_ae93f57743fa.slice/crio-9f33d30ab51d01d684e9a063d64e96b3903b0505cae218aa5f9e2d1a51918330 WatchSource:0}: Error finding container 9f33d30ab51d01d684e9a063d64e96b3903b0505cae218aa5f9e2d1a51918330: Status 404 returned error can't find the container with id 9f33d30ab51d01d684e9a063d64e96b3903b0505cae218aa5f9e2d1a51918330 Jul 14 19:41:20.588381 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.588350 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:41:20.588457 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.588421 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:41:20.588457 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:20.588450 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:41:21.576321 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:21.576278 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" event={"ID":"74e5ce24-18f9-4ae3-a604-ae93f57743fa","Type":"ContainerStarted","Data":"75ae929de90bc20c96cee25e190ba7f62f959a9234f00431c272db40d2740bcc"} Jul 14 19:41:21.576820 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:21.576348 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" event={"ID":"74e5ce24-18f9-4ae3-a604-ae93f57743fa","Type":"ContainerStarted","Data":"9f33d30ab51d01d684e9a063d64e96b3903b0505cae218aa5f9e2d1a51918330"} Jul 14 19:41:21.604167 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:21.604109 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" podStartSLOduration=1.604086211 podStartE2EDuration="1.604086211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:41:21.602231265 +0000 UTC m=+1834.715481136" watchObservedRunningTime="2026-07-14 19:41:21.604086211 +0000 UTC m=+1834.717336081" Jul 14 19:41:22.437230 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:22.437194 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:22.442070 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:22.442047 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:22.582061 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:22.582027 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:22.583020 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:22.582995 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:26.399739 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:26.399703 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-102a2069-zs75k"] Jul 14 19:41:26.403331 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:26.403314 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-102a2069-zs75k" Jul 14 19:41:26.406971 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:26.406949 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-br5f6\"" Jul 14 19:41:26.413211 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:26.413189 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-102a2069-zs75k"] Jul 14 19:41:26.440874 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:26.440838 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afc9462-76c0-42cc-9f56-edd8a76bec85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-102a2069-zs75k\" (UID: \"8afc9462-76c0-42cc-9f56-edd8a76bec85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-102a2069-zs75k" Jul 14 19:41:26.542040 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:26.542000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afc9462-76c0-42cc-9f56-edd8a76bec85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-102a2069-zs75k\" (UID: \"8afc9462-76c0-42cc-9f56-edd8a76bec85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-102a2069-zs75k" Jul 14 19:41:26.542697 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:26.542678 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afc9462-76c0-42cc-9f56-edd8a76bec85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-102a2069-zs75k\" (UID: \"8afc9462-76c0-42cc-9f56-edd8a76bec85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-102a2069-zs75k" Jul 14 19:41:26.714251 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:26.714178 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-102a2069-zs75k" Jul 14 19:41:26.846441 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:26.846408 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-102a2069-zs75k"] Jul 14 19:41:26.849240 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:41:26.849212 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afc9462_76c0_42cc_9f56_edd8a76bec85.slice/crio-f032599bd0e4f31027899450f7b4c405028f83154c5ccaf328b1013f88de079a WatchSource:0}: Error finding container f032599bd0e4f31027899450f7b4c405028f83154c5ccaf328b1013f88de079a: Status 404 returned error can't find the container with id f032599bd0e4f31027899450f7b4c405028f83154c5ccaf328b1013f88de079a Jul 14 19:41:26.851313 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:26.851292 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:41:27.607945 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:27.607910 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-102a2069-zs75k" event={"ID":"8afc9462-76c0-42cc-9f56-edd8a76bec85","Type":"ContainerStarted","Data":"f032599bd0e4f31027899450f7b4c405028f83154c5ccaf328b1013f88de079a"} Jul 14 19:41:29.621970 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:29.621931 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-102a2069-zs75k" event={"ID":"8afc9462-76c0-42cc-9f56-edd8a76bec85","Type":"ContainerStarted","Data":"ef02cc950f39d0b422caa81513e5a6b85ef020cf257cd0be86615491bee3ecc4"} Jul 14 19:41:29.639192 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:29.639130 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-102a2069-zs75k" podStartSLOduration=1.251652242 podStartE2EDuration="3.639113048s" totalImagesPullingTime="2.387460806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:41:29.636630639 +0000 UTC m=+1842.749880510" watchObservedRunningTime="2026-07-14 19:41:29.639113048 +0000 UTC m=+1842.752362918" Jul 14 19:41:30.628709 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:30.628673 2576 generic.go:356] "Generic (PLEG): container finished" podID="8afc9462-76c0-42cc-9f56-edd8a76bec85" containerID="ef02cc950f39d0b422caa81513e5a6b85ef020cf257cd0be86615491bee3ecc4" exitCode=0 Jul 14 19:41:30.629079 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:30.628742 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-102a2069-zs75k" event={"ID":"8afc9462-76c0-42cc-9f56-edd8a76bec85","Type":"ContainerDied","Data":"ef02cc950f39d0b422caa81513e5a6b85ef020cf257cd0be86615491bee3ecc4"} Jul 14 19:41:31.770229 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:31.770203 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-102a2069-zs75k" Jul 14 19:41:31.907276 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:31.907184 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afc9462-76c0-42cc-9f56-edd8a76bec85-maas-api-service-ca\") pod \"8afc9462-76c0-42cc-9f56-edd8a76bec85\" (UID: \"8afc9462-76c0-42cc-9f56-edd8a76bec85\") " Jul 14 19:41:31.907552 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:31.907528 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8afc9462-76c0-42cc-9f56-edd8a76bec85-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8afc9462-76c0-42cc-9f56-edd8a76bec85" (UID: "8afc9462-76c0-42cc-9f56-edd8a76bec85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:41:32.008335 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:32.008295 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afc9462-76c0-42cc-9f56-edd8a76bec85-maas-api-service-ca\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:41:32.640995 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:32.640961 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-102a2069-zs75k" event={"ID":"8afc9462-76c0-42cc-9f56-edd8a76bec85","Type":"ContainerDied","Data":"f032599bd0e4f31027899450f7b4c405028f83154c5ccaf328b1013f88de079a"} Jul 14 19:41:32.640995 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:32.640980 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-102a2069-zs75k" Jul 14 19:41:32.641207 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:32.641013 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f032599bd0e4f31027899450f7b4c405028f83154c5ccaf328b1013f88de079a" Jul 14 19:41:34.657772 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:34.657736 2576 generic.go:356] "Generic (PLEG): container finished" podID="74e5ce24-18f9-4ae3-a604-ae93f57743fa" containerID="75ae929de90bc20c96cee25e190ba7f62f959a9234f00431c272db40d2740bcc" exitCode=0 Jul 14 19:41:34.658192 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:34.657802 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" event={"ID":"74e5ce24-18f9-4ae3-a604-ae93f57743fa","Type":"ContainerDied","Data":"75ae929de90bc20c96cee25e190ba7f62f959a9234f00431c272db40d2740bcc"} Jul 14 19:41:34.658238 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:34.658206 2576 scope.go:117] "RemoveContainer" containerID="75ae929de90bc20c96cee25e190ba7f62f959a9234f00431c272db40d2740bcc" Jul 14 19:41:34.659233 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:34.659203 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:41:34.659332 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:34.659281 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:41:34.659402 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:34.659327 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:41:35.664099 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:35.664060 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" event={"ID":"74e5ce24-18f9-4ae3-a604-ae93f57743fa","Type":"ContainerStarted","Data":"78a7bfe958ac968675ca32f7cddf4e0b477624cd13df3dd669b4eaf1314d2fbd"} Jul 14 19:41:36.437758 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:36.437713 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:36.438992 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:36.438965 2576 patch_prober.go:28] interesting pod/e2e-ait-102a2069-openshift-default-684db9f89-425wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 14 19:41:36.439065 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:36.439032 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" podUID="74e5ce24-18f9-4ae3-a604-ae93f57743fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 14 19:41:37.438407 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:37.438371 2576 patch_prober.go:28] interesting pod/e2e-ait-102a2069-openshift-default-684db9f89-425wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 14 19:41:37.438835 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:37.438440 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" podUID="74e5ce24-18f9-4ae3-a604-ae93f57743fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 14 19:41:37.582072 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:37.582034 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:38.438282 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:38.438238 2576 patch_prober.go:28] interesting pod/e2e-ait-102a2069-openshift-default-684db9f89-425wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 14 19:41:38.438493 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:38.438293 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" podUID="74e5ce24-18f9-4ae3-a604-ae93f57743fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 14 19:41:39.437864 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:39.437823 2576 patch_prober.go:28] interesting pod/e2e-ait-102a2069-openshift-default-684db9f89-425wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 14 19:41:39.438054 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:39.437884 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" podUID="74e5ce24-18f9-4ae3-a604-ae93f57743fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 14 19:41:40.442064 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:40.442032 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:40.443070 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:40.443051 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:51.691058 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:51.691022 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl"] Jul 14 19:41:51.691576 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:51.691236 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" podUID="74e5ce24-18f9-4ae3-a604-ae93f57743fa" containerName="istio-proxy" containerID="cri-o://78a7bfe958ac968675ca32f7cddf4e0b477624cd13df3dd669b4eaf1314d2fbd" gracePeriod=30 Jul 14 19:41:52.357408 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.357377 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b"] Jul 14 19:41:52.357801 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.357787 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8afc9462-76c0-42cc-9f56-edd8a76bec85" containerName="revoke-keys" Jul 14 19:41:52.357801 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.357802 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8afc9462-76c0-42cc-9f56-edd8a76bec85" containerName="revoke-keys" Jul 14 19:41:52.357892 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.357857 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8afc9462-76c0-42cc-9f56-edd8a76bec85" containerName="revoke-keys" Jul 14 19:41:52.360943 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.360927 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.364943 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.364922 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-0bc32ee5-gw-openshift-default-dockercfg-hhxgc\"" Jul 14 19:41:52.387309 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.387279 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b"] Jul 14 19:41:52.416251 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.416219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-credential-socket\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.416446 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.416261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-data\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.416446 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.416284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-445d7\" (UniqueName: \"kubernetes.io/projected/9a8dcb17-d7e2-4999-9b55-857b79a86056-kube-api-access-445d7\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.416446 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.416314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-envoy\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.416446 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.416337 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-token\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.416446 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.416384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-podinfo\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.416446 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.416418 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8dcb17-d7e2-4999-9b55-857b79a86056-istiod-ca-cert\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.416712 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.416458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-workload-socket\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.416712 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.416485 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-workload-certs\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.516958 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.516928 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-445d7\" (UniqueName: \"kubernetes.io/projected/9a8dcb17-d7e2-4999-9b55-857b79a86056-kube-api-access-445d7\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.517159 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.516971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-envoy\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.517159 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.516989 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-token\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.517159 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.517044 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-podinfo\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.517159 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.517077 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8dcb17-d7e2-4999-9b55-857b79a86056-istiod-ca-cert\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.517159 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.517110 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-workload-socket\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.517159 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.517146 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-workload-certs\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.517495 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.517215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-credential-socket\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.517495 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.517255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-data\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.517625 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.517520 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-workload-socket\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.517625 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.517576 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-data\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.517739 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.517669 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-workload-certs\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.517878 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.517853 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-credential-socket\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.518027 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.517880 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8dcb17-d7e2-4999-9b55-857b79a86056-istiod-ca-cert\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.519606 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.519542 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-envoy\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.519793 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.519773 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-podinfo\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.525525 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.525504 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-445d7\" (UniqueName: \"kubernetes.io/projected/9a8dcb17-d7e2-4999-9b55-857b79a86056-kube-api-access-445d7\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.525652 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.525620 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-token\") pod \"e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.671528 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.671443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:52.821884 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.821860 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b"] Jul 14 19:41:52.824253 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:41:52.824225 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8dcb17_d7e2_4999_9b55_857b79a86056.slice/crio-3f48da83d10878dd88a0e15e1f061d9478c007f48d92763ce4317df1d6d426e4 WatchSource:0}: Error finding container 3f48da83d10878dd88a0e15e1f061d9478c007f48d92763ce4317df1d6d426e4: Status 404 returned error can't find the container with id 3f48da83d10878dd88a0e15e1f061d9478c007f48d92763ce4317df1d6d426e4 Jul 14 19:41:52.826354 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.826321 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:41:52.826422 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.826387 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:41:52.826422 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:52.826416 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:41:53.759908 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:53.759871 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" event={"ID":"9a8dcb17-d7e2-4999-9b55-857b79a86056","Type":"ContainerStarted","Data":"1e8e4bd70c173fa789dbbfbfb73a4f4ffe2452e14f17faeb7bc10fc42fe74607"} Jul 14 19:41:53.760113 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:53.759929 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" event={"ID":"9a8dcb17-d7e2-4999-9b55-857b79a86056","Type":"ContainerStarted","Data":"3f48da83d10878dd88a0e15e1f061d9478c007f48d92763ce4317df1d6d426e4"} Jul 14 19:41:53.782172 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:53.782113 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" podStartSLOduration=1.782091604 podStartE2EDuration="1.782091604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:41:53.780283859 +0000 UTC m=+1866.893533734" watchObservedRunningTime="2026-07-14 19:41:53.782091604 +0000 UTC m=+1866.895341475" Jul 14 19:41:54.672522 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:54.672480 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:54.677350 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:54.677323 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:54.765417 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:54.765388 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:54.766356 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:54.766336 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:41:55.444366 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:55.444325 2576 patch_prober.go:28] interesting pod/e2e-ait-102a2069-openshift-default-684db9f89-425wl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 19:41:55.444569 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:55.444406 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" podUID="74e5ce24-18f9-4ae3-a604-ae93f57743fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 19:41:56.946436 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.946410 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:56.963194 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.963109 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-token\") pod \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " Jul 14 19:41:56.963333 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.963263 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-workload-certs\") pod \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " Jul 14 19:41:56.963333 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.963294 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istiod-ca-cert\") pod \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " Jul 14 19:41:56.963333 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.963323 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-data\") pod \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " Jul 14 19:41:56.963484 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.963414 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-podinfo\") pod \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " Jul 14 19:41:56.963484 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.963480 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-workload-socket\") pod \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " Jul 14 19:41:56.963605 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.963519 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zs6b5\" (UniqueName: \"kubernetes.io/projected/74e5ce24-18f9-4ae3-a604-ae93f57743fa-kube-api-access-zs6b5\") pod \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " Jul 14 19:41:56.963605 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.963566 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-credential-socket\") pod \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " Jul 14 19:41:56.963980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.963609 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-envoy\") pod \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\" (UID: \"74e5ce24-18f9-4ae3-a604-ae93f57743fa\") " Jul 14 19:41:56.963980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.963756 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "74e5ce24-18f9-4ae3-a604-ae93f57743fa" (UID: "74e5ce24-18f9-4ae3-a604-ae93f57743fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:41:56.964120 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.963993 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istiod-ca-cert\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:41:56.964120 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.964029 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "74e5ce24-18f9-4ae3-a604-ae93f57743fa" (UID: "74e5ce24-18f9-4ae3-a604-ae93f57743fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:41:56.964226 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.964165 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "74e5ce24-18f9-4ae3-a604-ae93f57743fa" (UID: "74e5ce24-18f9-4ae3-a604-ae93f57743fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:41:56.964226 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.964195 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "74e5ce24-18f9-4ae3-a604-ae93f57743fa" (UID: "74e5ce24-18f9-4ae3-a604-ae93f57743fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:41:56.964329 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.964198 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "74e5ce24-18f9-4ae3-a604-ae93f57743fa" (UID: "74e5ce24-18f9-4ae3-a604-ae93f57743fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:41:56.965775 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.965743 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "74e5ce24-18f9-4ae3-a604-ae93f57743fa" (UID: "74e5ce24-18f9-4ae3-a604-ae93f57743fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:41:56.965888 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.965790 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "74e5ce24-18f9-4ae3-a604-ae93f57743fa" (UID: "74e5ce24-18f9-4ae3-a604-ae93f57743fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:41:56.965952 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.965895 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "74e5ce24-18f9-4ae3-a604-ae93f57743fa" (UID: "74e5ce24-18f9-4ae3-a604-ae93f57743fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 19:41:56.966174 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:56.966125 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e5ce24-18f9-4ae3-a604-ae93f57743fa-kube-api-access-zs6b5" (OuterVolumeSpecName: "kube-api-access-zs6b5") pod "74e5ce24-18f9-4ae3-a604-ae93f57743fa" (UID: "74e5ce24-18f9-4ae3-a604-ae93f57743fa"). InnerVolumeSpecName "kube-api-access-zs6b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:41:57.065005 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.064973 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-workload-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:41:57.065005 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.064999 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs6b5\" (UniqueName: \"kubernetes.io/projected/74e5ce24-18f9-4ae3-a604-ae93f57743fa-kube-api-access-zs6b5\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:41:57.065005 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.065010 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-credential-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:41:57.065217 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.065021 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-envoy\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:41:57.065217 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.065030 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-token\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:41:57.065217 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.065039 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-workload-certs\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:41:57.065217 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.065046 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-data\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:41:57.065217 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.065055 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e5ce24-18f9-4ae3-a604-ae93f57743fa-istio-podinfo\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:41:57.782495 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.782404 2576 generic.go:356] "Generic (PLEG): container finished" podID="74e5ce24-18f9-4ae3-a604-ae93f57743fa" containerID="78a7bfe958ac968675ca32f7cddf4e0b477624cd13df3dd669b4eaf1314d2fbd" exitCode=0 Jul 14 19:41:57.782495 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.782453 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" Jul 14 19:41:57.782495 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.782483 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" event={"ID":"74e5ce24-18f9-4ae3-a604-ae93f57743fa","Type":"ContainerDied","Data":"78a7bfe958ac968675ca32f7cddf4e0b477624cd13df3dd669b4eaf1314d2fbd"} Jul 14 19:41:57.782758 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.782528 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl" event={"ID":"74e5ce24-18f9-4ae3-a604-ae93f57743fa","Type":"ContainerDied","Data":"9f33d30ab51d01d684e9a063d64e96b3903b0505cae218aa5f9e2d1a51918330"} Jul 14 19:41:57.782758 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.782552 2576 scope.go:117] "RemoveContainer" containerID="78a7bfe958ac968675ca32f7cddf4e0b477624cd13df3dd669b4eaf1314d2fbd" Jul 14 19:41:57.791982 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.791960 2576 scope.go:117] "RemoveContainer" containerID="75ae929de90bc20c96cee25e190ba7f62f959a9234f00431c272db40d2740bcc" Jul 14 19:41:57.799862 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.799840 2576 scope.go:117] "RemoveContainer" containerID="78a7bfe958ac968675ca32f7cddf4e0b477624cd13df3dd669b4eaf1314d2fbd" Jul 14 19:41:57.800128 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:41:57.800108 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a7bfe958ac968675ca32f7cddf4e0b477624cd13df3dd669b4eaf1314d2fbd\": container with ID starting with 78a7bfe958ac968675ca32f7cddf4e0b477624cd13df3dd669b4eaf1314d2fbd not found: ID does not exist" containerID="78a7bfe958ac968675ca32f7cddf4e0b477624cd13df3dd669b4eaf1314d2fbd" Jul 14 19:41:57.800206 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.800140 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a7bfe958ac968675ca32f7cddf4e0b477624cd13df3dd669b4eaf1314d2fbd"} err="failed to get container status \"78a7bfe958ac968675ca32f7cddf4e0b477624cd13df3dd669b4eaf1314d2fbd\": rpc error: code = NotFound desc = could not find container \"78a7bfe958ac968675ca32f7cddf4e0b477624cd13df3dd669b4eaf1314d2fbd\": container with ID starting with 78a7bfe958ac968675ca32f7cddf4e0b477624cd13df3dd669b4eaf1314d2fbd not found: ID does not exist" Jul 14 19:41:57.800206 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.800165 2576 scope.go:117] "RemoveContainer" containerID="75ae929de90bc20c96cee25e190ba7f62f959a9234f00431c272db40d2740bcc" Jul 14 19:41:57.800434 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:41:57.800417 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ae929de90bc20c96cee25e190ba7f62f959a9234f00431c272db40d2740bcc\": container with ID starting with 75ae929de90bc20c96cee25e190ba7f62f959a9234f00431c272db40d2740bcc not found: ID does not exist" containerID="75ae929de90bc20c96cee25e190ba7f62f959a9234f00431c272db40d2740bcc" Jul 14 19:41:57.800472 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.800441 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ae929de90bc20c96cee25e190ba7f62f959a9234f00431c272db40d2740bcc"} err="failed to get container status \"75ae929de90bc20c96cee25e190ba7f62f959a9234f00431c272db40d2740bcc\": rpc error: code = NotFound desc = could not find container \"75ae929de90bc20c96cee25e190ba7f62f959a9234f00431c272db40d2740bcc\": container with ID starting with 75ae929de90bc20c96cee25e190ba7f62f959a9234f00431c272db40d2740bcc not found: ID does not exist" Jul 14 19:41:57.803776 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.803753 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl"] Jul 14 19:41:57.808203 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:57.808180 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-102a2069-openshift-default-684db9f89-425wl"] Jul 14 19:41:58.121351 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:58.121307 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc"] Jul 14 19:41:58.121843 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:58.121824 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74e5ce24-18f9-4ae3-a604-ae93f57743fa" containerName="istio-proxy" Jul 14 19:41:58.121900 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:58.121848 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="74e5ce24-18f9-4ae3-a604-ae93f57743fa" containerName="istio-proxy" Jul 14 19:41:58.121900 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:58.121862 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74e5ce24-18f9-4ae3-a604-ae93f57743fa" containerName="istio-proxy" Jul 14 19:41:58.121900 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:58.121871 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="74e5ce24-18f9-4ae3-a604-ae93f57743fa" containerName="istio-proxy" Jul 14 19:41:58.122010 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:58.121963 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="74e5ce24-18f9-4ae3-a604-ae93f57743fa" containerName="istio-proxy" Jul 14 19:41:58.126593 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:58.126564 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc" Jul 14 19:41:58.129792 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:58.129770 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-br5f6\"" Jul 14 19:41:58.132323 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:58.132298 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc"] Jul 14 19:41:58.175980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:58.175945 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54335b12-dfff-4cc9-9378-352099972dbc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc\" (UID: \"54335b12-dfff-4cc9-9378-352099972dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc" Jul 14 19:41:58.277264 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:58.277225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54335b12-dfff-4cc9-9378-352099972dbc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc\" (UID: \"54335b12-dfff-4cc9-9378-352099972dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc" Jul 14 19:41:58.278051 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:58.278026 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54335b12-dfff-4cc9-9378-352099972dbc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc\" (UID: \"54335b12-dfff-4cc9-9378-352099972dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc" Jul 14 19:41:58.440261 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:58.440180 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc" Jul 14 19:41:58.578758 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:58.578731 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc"] Jul 14 19:41:58.789475 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:58.789446 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc" event={"ID":"54335b12-dfff-4cc9-9378-352099972dbc","Type":"ContainerStarted","Data":"89bea5bcbc352e8f594d67b41a5bb781c7b046b18445038622ab1bd1543c8329"} Jul 14 19:41:59.516510 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:59.516427 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74e5ce24-18f9-4ae3-a604-ae93f57743fa" path="/var/lib/kubelet/pods/74e5ce24-18f9-4ae3-a604-ae93f57743fa/volumes" Jul 14 19:41:59.797466 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:59.797385 2576 generic.go:356] "Generic (PLEG): container finished" podID="54335b12-dfff-4cc9-9378-352099972dbc" containerID="2e9c34afac26d1a3d078aaa3194c07409cd6abaf57b418bc933cbe680736f159" exitCode=7 Jul 14 19:41:59.797466 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:59.797447 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc" event={"ID":"54335b12-dfff-4cc9-9378-352099972dbc","Type":"ContainerDied","Data":"2e9c34afac26d1a3d078aaa3194c07409cd6abaf57b418bc933cbe680736f159"} Jul 14 19:41:59.797690 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:41:59.797677 2576 scope.go:117] "RemoveContainer" containerID="2e9c34afac26d1a3d078aaa3194c07409cd6abaf57b418bc933cbe680736f159" Jul 14 19:42:00.806640 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:00.806600 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc" event={"ID":"54335b12-dfff-4cc9-9378-352099972dbc","Type":"ContainerStarted","Data":"987a1fc3c9a066abe5b6e9e2ca63f8e8ece34e8728cdc1df132ad7340ead3a0c"} Jul 14 19:42:00.822081 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:00.822028 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc" podStartSLOduration=2.822011391 podStartE2EDuration="2.822011391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:42:00.821676404 +0000 UTC m=+1873.934926292" watchObservedRunningTime="2026-07-14 19:42:00.822011391 +0000 UTC m=+1873.935261260" Jul 14 19:42:01.814190 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:01.814158 2576 generic.go:356] "Generic (PLEG): container finished" podID="54335b12-dfff-4cc9-9378-352099972dbc" containerID="987a1fc3c9a066abe5b6e9e2ca63f8e8ece34e8728cdc1df132ad7340ead3a0c" exitCode=0 Jul 14 19:42:01.814630 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:01.814199 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc" event={"ID":"54335b12-dfff-4cc9-9378-352099972dbc","Type":"ContainerDied","Data":"987a1fc3c9a066abe5b6e9e2ca63f8e8ece34e8728cdc1df132ad7340ead3a0c"} Jul 14 19:42:01.814630 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:01.814243 2576 scope.go:117] "RemoveContainer" containerID="2e9c34afac26d1a3d078aaa3194c07409cd6abaf57b418bc933cbe680736f159" Jul 14 19:42:02.972372 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:02.972101 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc" Jul 14 19:42:03.030443 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:03.030415 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54335b12-dfff-4cc9-9378-352099972dbc-maas-api-service-ca\") pod \"54335b12-dfff-4cc9-9378-352099972dbc\" (UID: \"54335b12-dfff-4cc9-9378-352099972dbc\") " Jul 14 19:42:03.030838 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:03.030801 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54335b12-dfff-4cc9-9378-352099972dbc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "54335b12-dfff-4cc9-9378-352099972dbc" (UID: "54335b12-dfff-4cc9-9378-352099972dbc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:42:03.131993 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:03.131907 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54335b12-dfff-4cc9-9378-352099972dbc-maas-api-service-ca\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:42:03.828008 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:03.827972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc" event={"ID":"54335b12-dfff-4cc9-9378-352099972dbc","Type":"ContainerDied","Data":"89bea5bcbc352e8f594d67b41a5bb781c7b046b18445038622ab1bd1543c8329"} Jul 14 19:42:03.828008 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:03.827993 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc" Jul 14 19:42:03.828218 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:03.828025 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89bea5bcbc352e8f594d67b41a5bb781c7b046b18445038622ab1bd1543c8329" Jul 14 19:42:07.850574 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:07.850539 2576 generic.go:356] "Generic (PLEG): container finished" podID="9a8dcb17-d7e2-4999-9b55-857b79a86056" containerID="1e8e4bd70c173fa789dbbfbfb73a4f4ffe2452e14f17faeb7bc10fc42fe74607" exitCode=0 Jul 14 19:42:07.851082 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:07.850612 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" event={"ID":"9a8dcb17-d7e2-4999-9b55-857b79a86056","Type":"ContainerDied","Data":"1e8e4bd70c173fa789dbbfbfb73a4f4ffe2452e14f17faeb7bc10fc42fe74607"} Jul 14 19:42:07.851082 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:07.851035 2576 scope.go:117] "RemoveContainer" containerID="1e8e4bd70c173fa789dbbfbfb73a4f4ffe2452e14f17faeb7bc10fc42fe74607" Jul 14 19:42:07.852095 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:07.852068 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:42:07.852181 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:07.852132 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:42:07.852181 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:07.852163 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:42:08.857554 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:08.857479 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" event={"ID":"9a8dcb17-d7e2-4999-9b55-857b79a86056","Type":"ContainerStarted","Data":"28e94e2d3de4e50c581bf0bd97025b6337d2240630cfa538fc7bac62614be926"} Jul 14 19:42:09.672403 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:09.672370 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:42:09.673296 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:09.673264 2576 patch_prober.go:28] interesting pod/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 14 19:42:09.673433 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:09.673329 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" podUID="9a8dcb17-d7e2-4999-9b55-857b79a86056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 14 19:42:09.765509 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:09.765481 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:42:10.671870 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:10.671827 2576 patch_prober.go:28] interesting pod/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 14 19:42:10.672521 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:10.672484 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" podUID="9a8dcb17-d7e2-4999-9b55-857b79a86056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 14 19:42:11.672667 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:11.672631 2576 patch_prober.go:28] interesting pod/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 14 19:42:11.673246 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:11.673212 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" podUID="9a8dcb17-d7e2-4999-9b55-857b79a86056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 14 19:42:12.672765 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:12.672723 2576 patch_prober.go:28] interesting pod/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 14 19:42:12.673401 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:12.673365 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" podUID="9a8dcb17-d7e2-4999-9b55-857b79a86056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 14 19:42:13.676069 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:13.676040 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:42:13.677345 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:13.677322 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:42:23.380405 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.380356 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b"] Jul 14 19:42:23.381055 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.380643 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" podUID="9a8dcb17-d7e2-4999-9b55-857b79a86056" containerName="istio-proxy" containerID="cri-o://28e94e2d3de4e50c581bf0bd97025b6337d2240630cfa538fc7bac62614be926" gracePeriod=30 Jul 14 19:42:23.689784 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.689703 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq"] Jul 14 19:42:23.690124 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.690111 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54335b12-dfff-4cc9-9378-352099972dbc" containerName="revoke-keys" Jul 14 19:42:23.690166 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.690126 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="54335b12-dfff-4cc9-9378-352099972dbc" containerName="revoke-keys" Jul 14 19:42:23.690166 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.690152 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54335b12-dfff-4cc9-9378-352099972dbc" containerName="revoke-keys" Jul 14 19:42:23.690166 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.690160 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="54335b12-dfff-4cc9-9378-352099972dbc" containerName="revoke-keys" Jul 14 19:42:23.690263 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.690224 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="54335b12-dfff-4cc9-9378-352099972dbc" containerName="revoke-keys" Jul 14 19:42:23.690263 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.690236 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="54335b12-dfff-4cc9-9378-352099972dbc" containerName="revoke-keys" Jul 14 19:42:23.690263 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.690245 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="74e5ce24-18f9-4ae3-a604-ae93f57743fa" containerName="istio-proxy" Jul 14 19:42:23.693330 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.693312 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.696257 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.696226 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f61c44f4-openshift-default-dockercfg-vq2t4\"" Jul 14 19:42:23.705182 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.705157 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq"] Jul 14 19:42:23.850035 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.849994 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-workload-socket\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.850035 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.850043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-envoy\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.850497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.850065 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-token\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.850497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.850173 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istiod-ca-cert\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.850497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.850218 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-credential-socket\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.850497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.850241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-podinfo\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.850497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.850304 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-workload-certs\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.850497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.850341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7bhn\" (UniqueName: \"kubernetes.io/projected/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-kube-api-access-k7bhn\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.850497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.850393 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-data\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.951120 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.951025 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-token\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.951120 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.951070 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istiod-ca-cert\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.951120 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.951090 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-credential-socket\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.951120 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.951109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-podinfo\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.951544 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.951157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-workload-certs\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.951544 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.951183 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bhn\" (UniqueName: \"kubernetes.io/projected/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-kube-api-access-k7bhn\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.951544 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.951207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-data\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.951544 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.951282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-workload-socket\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.951544 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.951319 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-envoy\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.951837 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.951766 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-workload-socket\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.951837 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.951801 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-workload-certs\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.951934 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.951835 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-data\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.951934 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.951900 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-credential-socket\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.952075 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.952055 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istiod-ca-cert\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.953726 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.953705 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-envoy\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.953850 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.953834 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-podinfo\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.959397 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.959374 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-token\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:23.959628 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:23.959610 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7bhn\" (UniqueName: \"kubernetes.io/projected/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-kube-api-access-k7bhn\") pod \"e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:24.006591 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:24.006540 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:24.157435 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:24.157408 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq"] Jul 14 19:42:24.159733 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:42:24.159703 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda486eb49_d6a8_426a_a8ad_e1ea13387d3a.slice/crio-fd3bc5cec121ee6b0e732716812483b4a35770ec412439f9395cc2b490f3b45d WatchSource:0}: Error finding container fd3bc5cec121ee6b0e732716812483b4a35770ec412439f9395cc2b490f3b45d: Status 404 returned error can't find the container with id fd3bc5cec121ee6b0e732716812483b4a35770ec412439f9395cc2b490f3b45d Jul 14 19:42:24.162235 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:24.162198 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:42:24.162345 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:24.162275 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:42:24.162345 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:24.162307 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:42:24.944721 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:24.944681 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" event={"ID":"a486eb49-d6a8-426a-a8ad-e1ea13387d3a","Type":"ContainerStarted","Data":"d6d9cf6bbf845155aef4ed6edc5c71ba7ed2d06adbdc334e7b706d429c9469ce"} Jul 14 19:42:24.945118 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:24.944737 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" event={"ID":"a486eb49-d6a8-426a-a8ad-e1ea13387d3a","Type":"ContainerStarted","Data":"fd3bc5cec121ee6b0e732716812483b4a35770ec412439f9395cc2b490f3b45d"} Jul 14 19:42:24.969941 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:24.969890 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" podStartSLOduration=1.9698716969999999 podStartE2EDuration="1.969871697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:42:24.969320046 +0000 UTC m=+1898.082569915" watchObservedRunningTime="2026-07-14 19:42:24.969871697 +0000 UTC m=+1898.083121546" Jul 14 19:42:25.007559 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:25.007515 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:25.012846 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:25.012822 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:25.950626 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:25.950592 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:25.951538 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:25.951520 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:28.701606 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.701057 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:42:28.798985 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.797912 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-data\") pod \"9a8dcb17-d7e2-4999-9b55-857b79a86056\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " Jul 14 19:42:28.798985 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.797978 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-credential-socket\") pod \"9a8dcb17-d7e2-4999-9b55-857b79a86056\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " Jul 14 19:42:28.798985 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.798013 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-podinfo\") pod \"9a8dcb17-d7e2-4999-9b55-857b79a86056\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " Jul 14 19:42:28.798985 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.798074 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-workload-socket\") pod \"9a8dcb17-d7e2-4999-9b55-857b79a86056\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " Jul 14 19:42:28.798985 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.798112 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-445d7\" (UniqueName: \"kubernetes.io/projected/9a8dcb17-d7e2-4999-9b55-857b79a86056-kube-api-access-445d7\") pod \"9a8dcb17-d7e2-4999-9b55-857b79a86056\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " Jul 14 19:42:28.798985 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.798138 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-workload-certs\") pod \"9a8dcb17-d7e2-4999-9b55-857b79a86056\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " Jul 14 19:42:28.798985 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.798166 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8dcb17-d7e2-4999-9b55-857b79a86056-istiod-ca-cert\") pod \"9a8dcb17-d7e2-4999-9b55-857b79a86056\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " Jul 14 19:42:28.798985 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.798219 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-token\") pod \"9a8dcb17-d7e2-4999-9b55-857b79a86056\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " Jul 14 19:42:28.798985 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.798244 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-envoy\") pod \"9a8dcb17-d7e2-4999-9b55-857b79a86056\" (UID: \"9a8dcb17-d7e2-4999-9b55-857b79a86056\") " Jul 14 19:42:28.801158 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.799888 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9a8dcb17-d7e2-4999-9b55-857b79a86056" (UID: "9a8dcb17-d7e2-4999-9b55-857b79a86056"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:42:28.801158 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.800661 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-data" (OuterVolumeSpecName: "istio-data") pod "9a8dcb17-d7e2-4999-9b55-857b79a86056" (UID: "9a8dcb17-d7e2-4999-9b55-857b79a86056"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:42:28.801158 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.800900 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9a8dcb17-d7e2-4999-9b55-857b79a86056" (UID: "9a8dcb17-d7e2-4999-9b55-857b79a86056"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:42:28.801667 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.801646 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9a8dcb17-d7e2-4999-9b55-857b79a86056" (UID: "9a8dcb17-d7e2-4999-9b55-857b79a86056"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:42:28.801978 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.801949 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8dcb17-d7e2-4999-9b55-857b79a86056-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9a8dcb17-d7e2-4999-9b55-857b79a86056" (UID: "9a8dcb17-d7e2-4999-9b55-857b79a86056"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:42:28.804951 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.804916 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9a8dcb17-d7e2-4999-9b55-857b79a86056" (UID: "9a8dcb17-d7e2-4999-9b55-857b79a86056"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:42:28.805676 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.805632 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-token" (OuterVolumeSpecName: "istio-token") pod "9a8dcb17-d7e2-4999-9b55-857b79a86056" (UID: "9a8dcb17-d7e2-4999-9b55-857b79a86056"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:42:28.805904 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.805872 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9a8dcb17-d7e2-4999-9b55-857b79a86056" (UID: "9a8dcb17-d7e2-4999-9b55-857b79a86056"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 19:42:28.806179 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.806147 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8dcb17-d7e2-4999-9b55-857b79a86056-kube-api-access-445d7" (OuterVolumeSpecName: "kube-api-access-445d7") pod "9a8dcb17-d7e2-4999-9b55-857b79a86056" (UID: "9a8dcb17-d7e2-4999-9b55-857b79a86056"). InnerVolumeSpecName "kube-api-access-445d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:42:28.899723 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.899675 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-data\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:42:28.899723 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.899722 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-credential-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:42:28.899723 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.899738 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-podinfo\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:42:28.900053 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.899753 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-workload-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:42:28.900053 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.899767 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-445d7\" (UniqueName: \"kubernetes.io/projected/9a8dcb17-d7e2-4999-9b55-857b79a86056-kube-api-access-445d7\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:42:28.900053 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.899780 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-workload-certs\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:42:28.900053 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.899792 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8dcb17-d7e2-4999-9b55-857b79a86056-istiod-ca-cert\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:42:28.900053 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.899806 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-token\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:42:28.900053 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.899818 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8dcb17-d7e2-4999-9b55-857b79a86056-istio-envoy\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:42:28.967093 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.966994 2576 generic.go:356] "Generic (PLEG): container finished" podID="9a8dcb17-d7e2-4999-9b55-857b79a86056" containerID="28e94e2d3de4e50c581bf0bd97025b6337d2240630cfa538fc7bac62614be926" exitCode=0 Jul 14 19:42:28.967093 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.967046 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" Jul 14 19:42:28.967093 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.967079 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" event={"ID":"9a8dcb17-d7e2-4999-9b55-857b79a86056","Type":"ContainerDied","Data":"28e94e2d3de4e50c581bf0bd97025b6337d2240630cfa538fc7bac62614be926"} Jul 14 19:42:28.967377 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.967142 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" event={"ID":"9a8dcb17-d7e2-4999-9b55-857b79a86056","Type":"ContainerDied","Data":"3f48da83d10878dd88a0e15e1f061d9478c007f48d92763ce4317df1d6d426e4"} Jul 14 19:42:28.967377 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.967167 2576 scope.go:117] "RemoveContainer" containerID="28e94e2d3de4e50c581bf0bd97025b6337d2240630cfa538fc7bac62614be926" Jul 14 19:42:28.982564 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:28.982388 2576 scope.go:117] "RemoveContainer" containerID="1e8e4bd70c173fa789dbbfbfb73a4f4ffe2452e14f17faeb7bc10fc42fe74607" Jul 14 19:42:29.004608 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:29.003999 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b"] Jul 14 19:42:29.006673 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:29.006621 2576 scope.go:117] "RemoveContainer" containerID="28e94e2d3de4e50c581bf0bd97025b6337d2240630cfa538fc7bac62614be926" Jul 14 19:42:29.007067 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:42:29.007044 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e94e2d3de4e50c581bf0bd97025b6337d2240630cfa538fc7bac62614be926\": container with ID starting with 28e94e2d3de4e50c581bf0bd97025b6337d2240630cfa538fc7bac62614be926 not found: ID does not exist" containerID="28e94e2d3de4e50c581bf0bd97025b6337d2240630cfa538fc7bac62614be926" Jul 14 19:42:29.007171 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:29.007081 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e94e2d3de4e50c581bf0bd97025b6337d2240630cfa538fc7bac62614be926"} err="failed to get container status \"28e94e2d3de4e50c581bf0bd97025b6337d2240630cfa538fc7bac62614be926\": rpc error: code = NotFound desc = could not find container \"28e94e2d3de4e50c581bf0bd97025b6337d2240630cfa538fc7bac62614be926\": container with ID starting with 28e94e2d3de4e50c581bf0bd97025b6337d2240630cfa538fc7bac62614be926 not found: ID does not exist" Jul 14 19:42:29.007171 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:29.007109 2576 scope.go:117] "RemoveContainer" containerID="1e8e4bd70c173fa789dbbfbfb73a4f4ffe2452e14f17faeb7bc10fc42fe74607" Jul 14 19:42:29.007469 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:42:29.007444 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8e4bd70c173fa789dbbfbfb73a4f4ffe2452e14f17faeb7bc10fc42fe74607\": container with ID starting with 1e8e4bd70c173fa789dbbfbfb73a4f4ffe2452e14f17faeb7bc10fc42fe74607 not found: ID does not exist" containerID="1e8e4bd70c173fa789dbbfbfb73a4f4ffe2452e14f17faeb7bc10fc42fe74607" Jul 14 19:42:29.007609 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:29.007475 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8e4bd70c173fa789dbbfbfb73a4f4ffe2452e14f17faeb7bc10fc42fe74607"} err="failed to get container status \"1e8e4bd70c173fa789dbbfbfb73a4f4ffe2452e14f17faeb7bc10fc42fe74607\": rpc error: code = NotFound desc = could not find container \"1e8e4bd70c173fa789dbbfbfb73a4f4ffe2452e14f17faeb7bc10fc42fe74607\": container with ID starting with 1e8e4bd70c173fa789dbbfbfb73a4f4ffe2452e14f17faeb7bc10fc42fe74607 not found: ID does not exist" Jul 14 19:42:29.008173 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:29.008151 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b"] Jul 14 19:42:29.515093 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:29.515061 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a8dcb17-d7e2-4999-9b55-857b79a86056" path="/var/lib/kubelet/pods/9a8dcb17-d7e2-4999-9b55-857b79a86056/volumes" Jul 14 19:42:29.677784 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:29.677733 2576 patch_prober.go:28] interesting pod/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 19:42:29.678072 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:29.677826 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-0bc32ee5-gw-openshift-default-c8666d6dc-s7w6b" podUID="9a8dcb17-d7e2-4999-9b55-857b79a86056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 19:42:30.005623 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:30.005573 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61c44f4-sk658"] Jul 14 19:42:30.006040 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:30.006021 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8dcb17-d7e2-4999-9b55-857b79a86056" containerName="istio-proxy" Jul 14 19:42:30.006040 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:30.006037 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8dcb17-d7e2-4999-9b55-857b79a86056" containerName="istio-proxy" Jul 14 19:42:30.006135 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:30.006062 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8dcb17-d7e2-4999-9b55-857b79a86056" containerName="istio-proxy" Jul 14 19:42:30.006135 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:30.006071 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8dcb17-d7e2-4999-9b55-857b79a86056" containerName="istio-proxy" Jul 14 19:42:30.006208 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:30.006151 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a8dcb17-d7e2-4999-9b55-857b79a86056" containerName="istio-proxy" Jul 14 19:42:30.006208 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:30.006163 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a8dcb17-d7e2-4999-9b55-857b79a86056" containerName="istio-proxy" Jul 14 19:42:30.011098 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:30.011074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61c44f4-sk658" Jul 14 19:42:30.013876 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:30.013845 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-br5f6\"" Jul 14 19:42:30.014638 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:30.014611 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61c44f4-sk658"] Jul 14 19:42:30.112117 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:30.112085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c15e78b-6940-4143-a1b2-e3cf4ae30c3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f61c44f4-sk658\" (UID: \"9c15e78b-6940-4143-a1b2-e3cf4ae30c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61c44f4-sk658" Jul 14 19:42:30.212725 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:30.212690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c15e78b-6940-4143-a1b2-e3cf4ae30c3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f61c44f4-sk658\" (UID: \"9c15e78b-6940-4143-a1b2-e3cf4ae30c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61c44f4-sk658" Jul 14 19:42:30.213376 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:30.213354 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c15e78b-6940-4143-a1b2-e3cf4ae30c3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f61c44f4-sk658\" (UID: \"9c15e78b-6940-4143-a1b2-e3cf4ae30c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61c44f4-sk658" Jul 14 19:42:30.324924 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:30.324871 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61c44f4-sk658" Jul 14 19:42:30.505753 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:30.505722 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61c44f4-sk658"] Jul 14 19:42:30.507952 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:42:30.507917 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c15e78b_6940_4143_a1b2_e3cf4ae30c3f.slice/crio-90ea277e922cb8bad7a5465228201a8a5633e7d476b7ea265d8a83620bbe7163 WatchSource:0}: Error finding container 90ea277e922cb8bad7a5465228201a8a5633e7d476b7ea265d8a83620bbe7163: Status 404 returned error can't find the container with id 90ea277e922cb8bad7a5465228201a8a5633e7d476b7ea265d8a83620bbe7163 Jul 14 19:42:30.989862 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:30.989439 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61c44f4-sk658" event={"ID":"9c15e78b-6940-4143-a1b2-e3cf4ae30c3f","Type":"ContainerStarted","Data":"506151c5119b699bd7888ba8aed928799c72449338c82753d558ab800e9e058e"} Jul 14 19:42:30.989862 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:30.989513 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61c44f4-sk658" event={"ID":"9c15e78b-6940-4143-a1b2-e3cf4ae30c3f","Type":"ContainerStarted","Data":"90ea277e922cb8bad7a5465228201a8a5633e7d476b7ea265d8a83620bbe7163"} Jul 14 19:42:31.009726 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:31.009663 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61c44f4-sk658" podStartSLOduration=2.00964366 podStartE2EDuration="2.00964366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:42:31.008082415 +0000 UTC m=+1904.121332286" watchObservedRunningTime="2026-07-14 19:42:31.00964366 +0000 UTC m=+1904.122893606" Jul 14 19:42:32.002274 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:32.002226 2576 generic.go:356] "Generic (PLEG): container finished" podID="9c15e78b-6940-4143-a1b2-e3cf4ae30c3f" containerID="506151c5119b699bd7888ba8aed928799c72449338c82753d558ab800e9e058e" exitCode=7 Jul 14 19:42:32.002469 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:32.002376 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61c44f4-sk658" event={"ID":"9c15e78b-6940-4143-a1b2-e3cf4ae30c3f","Type":"ContainerDied","Data":"506151c5119b699bd7888ba8aed928799c72449338c82753d558ab800e9e058e"} Jul 14 19:42:32.002813 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:32.002789 2576 scope.go:117] "RemoveContainer" containerID="506151c5119b699bd7888ba8aed928799c72449338c82753d558ab800e9e058e" Jul 14 19:42:33.010340 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:33.010307 2576 generic.go:356] "Generic (PLEG): container finished" podID="9c15e78b-6940-4143-a1b2-e3cf4ae30c3f" containerID="790b286584d6761d8cf908c8e11982054ab5ab374e0f2877d89dea3a12434c78" exitCode=0 Jul 14 19:42:33.010760 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:33.010389 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61c44f4-sk658" event={"ID":"9c15e78b-6940-4143-a1b2-e3cf4ae30c3f","Type":"ContainerDied","Data":"790b286584d6761d8cf908c8e11982054ab5ab374e0f2877d89dea3a12434c78"} Jul 14 19:42:33.010760 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:33.010438 2576 scope.go:117] "RemoveContainer" containerID="506151c5119b699bd7888ba8aed928799c72449338c82753d558ab800e9e058e" Jul 14 19:42:34.167469 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:34.167439 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61c44f4-sk658" Jul 14 19:42:34.261781 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:34.261749 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c15e78b-6940-4143-a1b2-e3cf4ae30c3f-maas-api-service-ca\") pod \"9c15e78b-6940-4143-a1b2-e3cf4ae30c3f\" (UID: \"9c15e78b-6940-4143-a1b2-e3cf4ae30c3f\") " Jul 14 19:42:34.262132 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:34.262109 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c15e78b-6940-4143-a1b2-e3cf4ae30c3f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9c15e78b-6940-4143-a1b2-e3cf4ae30c3f" (UID: "9c15e78b-6940-4143-a1b2-e3cf4ae30c3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:42:34.363451 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:34.363409 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c15e78b-6940-4143-a1b2-e3cf4ae30c3f-maas-api-service-ca\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:42:35.023160 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:35.023113 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61c44f4-sk658" event={"ID":"9c15e78b-6940-4143-a1b2-e3cf4ae30c3f","Type":"ContainerDied","Data":"90ea277e922cb8bad7a5465228201a8a5633e7d476b7ea265d8a83620bbe7163"} Jul 14 19:42:35.023160 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:35.023163 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61c44f4-sk658" Jul 14 19:42:35.023379 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:35.023177 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90ea277e922cb8bad7a5465228201a8a5633e7d476b7ea265d8a83620bbe7163" Jul 14 19:42:36.030285 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:36.030253 2576 generic.go:356] "Generic (PLEG): container finished" podID="a486eb49-d6a8-426a-a8ad-e1ea13387d3a" containerID="d6d9cf6bbf845155aef4ed6edc5c71ba7ed2d06adbdc334e7b706d429c9469ce" exitCode=0 Jul 14 19:42:36.030747 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:36.030304 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" event={"ID":"a486eb49-d6a8-426a-a8ad-e1ea13387d3a","Type":"ContainerDied","Data":"d6d9cf6bbf845155aef4ed6edc5c71ba7ed2d06adbdc334e7b706d429c9469ce"} Jul 14 19:42:36.030747 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:36.030737 2576 scope.go:117] "RemoveContainer" containerID="d6d9cf6bbf845155aef4ed6edc5c71ba7ed2d06adbdc334e7b706d429c9469ce" Jul 14 19:42:36.031935 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:36.031902 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:42:36.032005 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:36.031988 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:42:36.032050 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:36.032041 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:42:37.037855 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:37.037815 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" event={"ID":"a486eb49-d6a8-426a-a8ad-e1ea13387d3a","Type":"ContainerStarted","Data":"b0d6730196e71c0e11b9df1b0e61b42bc4a778ebb1860165a83171e2510002b7"} Jul 14 19:42:38.007781 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:38.007743 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:38.008898 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:38.008873 2576 patch_prober.go:28] interesting pod/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 14 19:42:38.009024 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:38.008921 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" podUID="a486eb49-d6a8-426a-a8ad-e1ea13387d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 14 19:42:39.007675 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:39.007638 2576 patch_prober.go:28] interesting pod/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 14 19:42:39.008074 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:39.007724 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" podUID="a486eb49-d6a8-426a-a8ad-e1ea13387d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 14 19:42:40.007754 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:40.007716 2576 patch_prober.go:28] interesting pod/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 14 19:42:40.008211 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:40.007776 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" podUID="a486eb49-d6a8-426a-a8ad-e1ea13387d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 14 19:42:40.950909 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:40.950868 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:41.007866 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:41.007826 2576 patch_prober.go:28] interesting pod/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 14 19:42:41.008276 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:41.007898 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" podUID="a486eb49-d6a8-426a-a8ad-e1ea13387d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 14 19:42:42.007481 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:42.007438 2576 patch_prober.go:28] interesting pod/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 14 19:42:42.007681 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:42.007505 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" podUID="a486eb49-d6a8-426a-a8ad-e1ea13387d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 14 19:42:43.011516 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:43.011486 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:43.012515 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:43.012495 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:42:56.205004 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.204970 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq"] Jul 14 19:42:56.205463 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.205187 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" podUID="a486eb49-d6a8-426a-a8ad-e1ea13387d3a" containerName="istio-proxy" containerID="cri-o://b0d6730196e71c0e11b9df1b0e61b42bc4a778ebb1860165a83171e2510002b7" gracePeriod=30 Jul 14 19:42:56.631805 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.631768 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf"] Jul 14 19:42:56.632173 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.632161 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c15e78b-6940-4143-a1b2-e3cf4ae30c3f" containerName="revoke-keys" Jul 14 19:42:56.632226 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.632175 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c15e78b-6940-4143-a1b2-e3cf4ae30c3f" containerName="revoke-keys" Jul 14 19:42:56.632226 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.632185 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c15e78b-6940-4143-a1b2-e3cf4ae30c3f" containerName="revoke-keys" Jul 14 19:42:56.632226 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.632190 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c15e78b-6940-4143-a1b2-e3cf4ae30c3f" containerName="revoke-keys" Jul 14 19:42:56.632319 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.632256 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c15e78b-6940-4143-a1b2-e3cf4ae30c3f" containerName="revoke-keys" Jul 14 19:42:56.632319 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.632265 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c15e78b-6940-4143-a1b2-e3cf4ae30c3f" containerName="revoke-keys" Jul 14 19:42:56.635488 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.635459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.639717 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.639692 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-9d072882-openshift-default-dockercfg-qkdc2\"" Jul 14 19:42:56.667462 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.667429 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf"] Jul 14 19:42:56.792997 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.792959 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzw5r\" (UniqueName: \"kubernetes.io/projected/f8b5510c-37c3-4935-a9fa-bf860b096000-kube-api-access-pzw5r\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.793181 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.793011 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-envoy\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.793181 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.793099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-podinfo\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.793181 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.793151 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-credential-socket\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.793303 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.793206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-workload-certs\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.793303 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.793238 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b5510c-37c3-4935-a9fa-bf860b096000-istiod-ca-cert\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.793379 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.793305 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-data\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.793379 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.793339 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-workload-socket\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.793379 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.793360 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-token\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.894196 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.894098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-credential-socket\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.894196 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.894140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-workload-certs\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.894409 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.894284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b5510c-37c3-4935-a9fa-bf860b096000-istiod-ca-cert\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.894409 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.894347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-data\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.894409 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.894377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-workload-socket\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.894542 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.894432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-token\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.894542 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.894462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzw5r\" (UniqueName: \"kubernetes.io/projected/f8b5510c-37c3-4935-a9fa-bf860b096000-kube-api-access-pzw5r\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.894542 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.894511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-envoy\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.894751 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.894550 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-credential-socket\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.894751 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.894573 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-workload-certs\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.894751 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.894614 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-podinfo\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.894751 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.894702 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-data\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.894931 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.894799 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-workload-socket\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.895049 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.895031 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b5510c-37c3-4935-a9fa-bf860b096000-istiod-ca-cert\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.896769 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.896748 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-envoy\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.896973 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.896955 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-podinfo\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.904139 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.904116 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-token\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.904237 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.904196 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzw5r\" (UniqueName: \"kubernetes.io/projected/f8b5510c-37c3-4935-a9fa-bf860b096000-kube-api-access-pzw5r\") pod \"e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:56.947520 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:56.947201 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:57.095003 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:57.094977 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf"] Jul 14 19:42:57.097296 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:42:57.097268 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b5510c_37c3_4935_a9fa_bf860b096000.slice/crio-24be63ed63cab75f0e54d8d2133065a7589611c1b3dc87b4b8791526def1a010 WatchSource:0}: Error finding container 24be63ed63cab75f0e54d8d2133065a7589611c1b3dc87b4b8791526def1a010: Status 404 returned error can't find the container with id 24be63ed63cab75f0e54d8d2133065a7589611c1b3dc87b4b8791526def1a010 Jul 14 19:42:57.099517 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:57.099486 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:42:57.099626 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:57.099553 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:42:57.099626 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:57.099596 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:42:57.136356 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:57.136326 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" event={"ID":"f8b5510c-37c3-4935-a9fa-bf860b096000","Type":"ContainerStarted","Data":"24be63ed63cab75f0e54d8d2133065a7589611c1b3dc87b4b8791526def1a010"} Jul 14 19:42:58.014871 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:58.014836 2576 patch_prober.go:28] interesting pod/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 19:42:58.015412 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:58.014899 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" podUID="a486eb49-d6a8-426a-a8ad-e1ea13387d3a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 19:42:58.144072 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:58.144032 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" event={"ID":"f8b5510c-37c3-4935-a9fa-bf860b096000","Type":"ContainerStarted","Data":"c572aebd79a88aa086015e547acd260ffd24448def4d662f7feee6254957ed56"} Jul 14 19:42:58.176914 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:58.176838 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" podStartSLOduration=2.176813677 podStartE2EDuration="2.176813677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:42:58.170891079 +0000 UTC m=+1931.284140949" watchObservedRunningTime="2026-07-14 19:42:58.176813677 +0000 UTC m=+1931.290063547" Jul 14 19:42:58.947676 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:58.947635 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:58.952897 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:58.952869 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:59.149696 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:59.149664 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:42:59.150643 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:42:59.150624 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:43:01.463700 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.463666 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:43:01.540597 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.540545 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7bhn\" (UniqueName: \"kubernetes.io/projected/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-kube-api-access-k7bhn\") pod \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " Jul 14 19:43:01.540771 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.540625 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-workload-socket\") pod \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " Jul 14 19:43:01.540771 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.540660 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-envoy\") pod \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " Jul 14 19:43:01.540771 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.540686 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-token\") pod \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " Jul 14 19:43:01.540771 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.540752 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-data\") pod \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " Jul 14 19:43:01.540995 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.540793 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istiod-ca-cert\") pod \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " Jul 14 19:43:01.540995 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.540831 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-workload-certs\") pod \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " Jul 14 19:43:01.540995 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.540899 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-credential-socket\") pod \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " Jul 14 19:43:01.540995 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.540924 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a486eb49-d6a8-426a-a8ad-e1ea13387d3a" (UID: "a486eb49-d6a8-426a-a8ad-e1ea13387d3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:43:01.540995 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.540937 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-podinfo\") pod \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\" (UID: \"a486eb49-d6a8-426a-a8ad-e1ea13387d3a\") " Jul 14 19:43:01.541373 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.541352 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-workload-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:43:01.541709 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.541494 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a486eb49-d6a8-426a-a8ad-e1ea13387d3a" (UID: "a486eb49-d6a8-426a-a8ad-e1ea13387d3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:43:01.541831 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.541570 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a486eb49-d6a8-426a-a8ad-e1ea13387d3a" (UID: "a486eb49-d6a8-426a-a8ad-e1ea13387d3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:43:01.541831 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.541675 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a486eb49-d6a8-426a-a8ad-e1ea13387d3a" (UID: "a486eb49-d6a8-426a-a8ad-e1ea13387d3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:43:01.541831 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.541683 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-data" (OuterVolumeSpecName: "istio-data") pod "a486eb49-d6a8-426a-a8ad-e1ea13387d3a" (UID: "a486eb49-d6a8-426a-a8ad-e1ea13387d3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:43:01.542975 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.542941 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-token" (OuterVolumeSpecName: "istio-token") pod "a486eb49-d6a8-426a-a8ad-e1ea13387d3a" (UID: "a486eb49-d6a8-426a-a8ad-e1ea13387d3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:43:01.543080 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.542984 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-kube-api-access-k7bhn" (OuterVolumeSpecName: "kube-api-access-k7bhn") pod "a486eb49-d6a8-426a-a8ad-e1ea13387d3a" (UID: "a486eb49-d6a8-426a-a8ad-e1ea13387d3a"). InnerVolumeSpecName "kube-api-access-k7bhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:43:01.543174 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.543152 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a486eb49-d6a8-426a-a8ad-e1ea13387d3a" (UID: "a486eb49-d6a8-426a-a8ad-e1ea13387d3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:43:01.543489 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.543468 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a486eb49-d6a8-426a-a8ad-e1ea13387d3a" (UID: "a486eb49-d6a8-426a-a8ad-e1ea13387d3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 19:43:01.641924 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.641884 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7bhn\" (UniqueName: \"kubernetes.io/projected/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-kube-api-access-k7bhn\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:43:01.641924 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.641920 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-envoy\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:43:01.641924 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.641929 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-token\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:43:01.642169 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.641937 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-data\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:43:01.642169 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.641946 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istiod-ca-cert\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:43:01.642169 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.641955 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-workload-certs\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:43:01.642169 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.641963 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-credential-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:43:01.642169 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:01.641971 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a486eb49-d6a8-426a-a8ad-e1ea13387d3a-istio-podinfo\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:43:02.157657 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.157621 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n"] Jul 14 19:43:02.158052 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.158038 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a486eb49-d6a8-426a-a8ad-e1ea13387d3a" containerName="istio-proxy" Jul 14 19:43:02.158093 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.158054 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a486eb49-d6a8-426a-a8ad-e1ea13387d3a" containerName="istio-proxy" Jul 14 19:43:02.158093 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.158065 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a486eb49-d6a8-426a-a8ad-e1ea13387d3a" containerName="istio-proxy" Jul 14 19:43:02.158093 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.158074 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a486eb49-d6a8-426a-a8ad-e1ea13387d3a" containerName="istio-proxy" Jul 14 19:43:02.158200 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.158160 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a486eb49-d6a8-426a-a8ad-e1ea13387d3a" containerName="istio-proxy" Jul 14 19:43:02.158200 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.158169 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a486eb49-d6a8-426a-a8ad-e1ea13387d3a" containerName="istio-proxy" Jul 14 19:43:02.161569 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.161535 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n" Jul 14 19:43:02.165460 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.165437 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-br5f6\"" Jul 14 19:43:02.168559 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.168506 2576 generic.go:356] "Generic (PLEG): container finished" podID="a486eb49-d6a8-426a-a8ad-e1ea13387d3a" containerID="b0d6730196e71c0e11b9df1b0e61b42bc4a778ebb1860165a83171e2510002b7" exitCode=0 Jul 14 19:43:02.168708 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.168576 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" Jul 14 19:43:02.168708 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.168649 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" event={"ID":"a486eb49-d6a8-426a-a8ad-e1ea13387d3a","Type":"ContainerDied","Data":"b0d6730196e71c0e11b9df1b0e61b42bc4a778ebb1860165a83171e2510002b7"} Jul 14 19:43:02.168708 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.168700 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq" event={"ID":"a486eb49-d6a8-426a-a8ad-e1ea13387d3a","Type":"ContainerDied","Data":"fd3bc5cec121ee6b0e732716812483b4a35770ec412439f9395cc2b490f3b45d"} Jul 14 19:43:02.168884 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.168732 2576 scope.go:117] "RemoveContainer" containerID="b0d6730196e71c0e11b9df1b0e61b42bc4a778ebb1860165a83171e2510002b7" Jul 14 19:43:02.172030 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.171965 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n"] Jul 14 19:43:02.182666 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.182631 2576 scope.go:117] "RemoveContainer" containerID="d6d9cf6bbf845155aef4ed6edc5c71ba7ed2d06adbdc334e7b706d429c9469ce" Jul 14 19:43:02.191787 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.191722 2576 scope.go:117] "RemoveContainer" containerID="b0d6730196e71c0e11b9df1b0e61b42bc4a778ebb1860165a83171e2510002b7" Jul 14 19:43:02.192050 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:43:02.192030 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d6730196e71c0e11b9df1b0e61b42bc4a778ebb1860165a83171e2510002b7\": container with ID starting with b0d6730196e71c0e11b9df1b0e61b42bc4a778ebb1860165a83171e2510002b7 not found: ID does not exist" containerID="b0d6730196e71c0e11b9df1b0e61b42bc4a778ebb1860165a83171e2510002b7" Jul 14 19:43:02.192132 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.192059 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d6730196e71c0e11b9df1b0e61b42bc4a778ebb1860165a83171e2510002b7"} err="failed to get container status \"b0d6730196e71c0e11b9df1b0e61b42bc4a778ebb1860165a83171e2510002b7\": rpc error: code = NotFound desc = could not find container \"b0d6730196e71c0e11b9df1b0e61b42bc4a778ebb1860165a83171e2510002b7\": container with ID starting with b0d6730196e71c0e11b9df1b0e61b42bc4a778ebb1860165a83171e2510002b7 not found: ID does not exist" Jul 14 19:43:02.192132 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.192077 2576 scope.go:117] "RemoveContainer" containerID="d6d9cf6bbf845155aef4ed6edc5c71ba7ed2d06adbdc334e7b706d429c9469ce" Jul 14 19:43:02.192338 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:43:02.192317 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d9cf6bbf845155aef4ed6edc5c71ba7ed2d06adbdc334e7b706d429c9469ce\": container with ID starting with d6d9cf6bbf845155aef4ed6edc5c71ba7ed2d06adbdc334e7b706d429c9469ce not found: ID does not exist" containerID="d6d9cf6bbf845155aef4ed6edc5c71ba7ed2d06adbdc334e7b706d429c9469ce" Jul 14 19:43:02.192402 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.192345 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d9cf6bbf845155aef4ed6edc5c71ba7ed2d06adbdc334e7b706d429c9469ce"} err="failed to get container status \"d6d9cf6bbf845155aef4ed6edc5c71ba7ed2d06adbdc334e7b706d429c9469ce\": rpc error: code = NotFound desc = could not find container \"d6d9cf6bbf845155aef4ed6edc5c71ba7ed2d06adbdc334e7b706d429c9469ce\": container with ID starting with d6d9cf6bbf845155aef4ed6edc5c71ba7ed2d06adbdc334e7b706d429c9469ce not found: ID does not exist" Jul 14 19:43:02.199982 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.199954 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq"] Jul 14 19:43:02.205853 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.205828 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f61c44f4-openshift-default-68f7fc99f-98cqq"] Jul 14 19:43:02.246568 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.246534 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfa7542-a27e-49af-9e0d-960a9057f3c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9d072882-xrr5n\" (UID: \"adfa7542-a27e-49af-9e0d-960a9057f3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n" Jul 14 19:43:02.347102 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.347061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfa7542-a27e-49af-9e0d-960a9057f3c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9d072882-xrr5n\" (UID: \"adfa7542-a27e-49af-9e0d-960a9057f3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n" Jul 14 19:43:02.347736 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.347715 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfa7542-a27e-49af-9e0d-960a9057f3c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9d072882-xrr5n\" (UID: \"adfa7542-a27e-49af-9e0d-960a9057f3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n" Jul 14 19:43:02.475477 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.475370 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n" Jul 14 19:43:02.643288 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:02.643255 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n"] Jul 14 19:43:02.644863 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:43:02.644834 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfa7542_a27e_49af_9e0d_960a9057f3c9.slice/crio-ec29835083b18a656d39db48f1fba98a4bd81d05d5c65b91091bca707983af90 WatchSource:0}: Error finding container ec29835083b18a656d39db48f1fba98a4bd81d05d5c65b91091bca707983af90: Status 404 returned error can't find the container with id ec29835083b18a656d39db48f1fba98a4bd81d05d5c65b91091bca707983af90 Jul 14 19:43:03.179272 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:03.179162 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n" event={"ID":"adfa7542-a27e-49af-9e0d-960a9057f3c9","Type":"ContainerStarted","Data":"a4bb892d057dce0887275e9cf8f450ff9e493c075d9ae523ab6e348c0404f52e"} Jul 14 19:43:03.179272 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:03.179225 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n" event={"ID":"adfa7542-a27e-49af-9e0d-960a9057f3c9","Type":"ContainerStarted","Data":"ec29835083b18a656d39db48f1fba98a4bd81d05d5c65b91091bca707983af90"} Jul 14 19:43:03.218847 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:03.218793 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n" podStartSLOduration=1.218777723 podStartE2EDuration="1.218777723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:43:03.215698624 +0000 UTC m=+1936.328948491" watchObservedRunningTime="2026-07-14 19:43:03.218777723 +0000 UTC m=+1936.332027678" Jul 14 19:43:03.516027 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:03.515946 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a486eb49-d6a8-426a-a8ad-e1ea13387d3a" path="/var/lib/kubelet/pods/a486eb49-d6a8-426a-a8ad-e1ea13387d3a/volumes" Jul 14 19:43:04.190861 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:04.190823 2576 generic.go:356] "Generic (PLEG): container finished" podID="adfa7542-a27e-49af-9e0d-960a9057f3c9" containerID="a4bb892d057dce0887275e9cf8f450ff9e493c075d9ae523ab6e348c0404f52e" exitCode=7 Jul 14 19:43:04.191179 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:04.191157 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n" event={"ID":"adfa7542-a27e-49af-9e0d-960a9057f3c9","Type":"ContainerDied","Data":"a4bb892d057dce0887275e9cf8f450ff9e493c075d9ae523ab6e348c0404f52e"} Jul 14 19:43:04.191651 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:04.191633 2576 scope.go:117] "RemoveContainer" containerID="a4bb892d057dce0887275e9cf8f450ff9e493c075d9ae523ab6e348c0404f52e" Jul 14 19:43:05.201313 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:05.201271 2576 generic.go:356] "Generic (PLEG): container finished" podID="adfa7542-a27e-49af-9e0d-960a9057f3c9" containerID="1a2ce54b2cd8403e80c574a3195dfa164fa8cac9c71b533bce677c1b50fcffb5" exitCode=7 Jul 14 19:43:05.201313 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:05.201306 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n" event={"ID":"adfa7542-a27e-49af-9e0d-960a9057f3c9","Type":"ContainerDied","Data":"1a2ce54b2cd8403e80c574a3195dfa164fa8cac9c71b533bce677c1b50fcffb5"} Jul 14 19:43:05.201918 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:05.201377 2576 scope.go:117] "RemoveContainer" containerID="a4bb892d057dce0887275e9cf8f450ff9e493c075d9ae523ab6e348c0404f52e" Jul 14 19:43:05.201918 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:05.201692 2576 scope.go:117] "RemoveContainer" containerID="1a2ce54b2cd8403e80c574a3195dfa164fa8cac9c71b533bce677c1b50fcffb5" Jul 14 19:43:05.202033 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:43:05.201979 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-9d072882-xrr5n_odh-ai-gateway-infra(adfa7542-a27e-49af-9e0d-960a9057f3c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n" podUID="adfa7542-a27e-49af-9e0d-960a9057f3c9" Jul 14 19:43:06.208226 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:06.208201 2576 scope.go:117] "RemoveContainer" containerID="1a2ce54b2cd8403e80c574a3195dfa164fa8cac9c71b533bce677c1b50fcffb5" Jul 14 19:43:06.208677 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:43:06.208396 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-9d072882-xrr5n_odh-ai-gateway-infra(adfa7542-a27e-49af-9e0d-960a9057f3c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n" podUID="adfa7542-a27e-49af-9e0d-960a9057f3c9" Jul 14 19:43:11.236504 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:11.236471 2576 generic.go:356] "Generic (PLEG): container finished" podID="f8b5510c-37c3-4935-a9fa-bf860b096000" containerID="c572aebd79a88aa086015e547acd260ffd24448def4d662f7feee6254957ed56" exitCode=0 Jul 14 19:43:11.236930 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:11.236552 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" event={"ID":"f8b5510c-37c3-4935-a9fa-bf860b096000","Type":"ContainerDied","Data":"c572aebd79a88aa086015e547acd260ffd24448def4d662f7feee6254957ed56"} Jul 14 19:43:11.237056 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:11.237041 2576 scope.go:117] "RemoveContainer" containerID="c572aebd79a88aa086015e547acd260ffd24448def4d662f7feee6254957ed56" Jul 14 19:43:11.238097 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:11.238058 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:43:11.238205 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:11.238135 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:43:11.238205 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:11.238180 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:43:12.243533 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:12.243499 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" event={"ID":"f8b5510c-37c3-4935-a9fa-bf860b096000","Type":"ContainerStarted","Data":"a48622ad3949312beba9f1b15b86ce0f6cfb8d7f5d122dd2d3e622e952b5f76f"} Jul 14 19:43:12.947659 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:12.947628 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:43:12.948593 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:12.948560 2576 patch_prober.go:28] interesting pod/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 14 19:43:12.948666 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:12.948634 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" podUID="f8b5510c-37c3-4935-a9fa-bf860b096000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 14 19:43:13.947784 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:13.947749 2576 patch_prober.go:28] interesting pod/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 14 19:43:13.948172 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:13.947811 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" podUID="f8b5510c-37c3-4935-a9fa-bf860b096000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 14 19:43:14.150174 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:14.150140 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:43:14.948595 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:14.948557 2576 patch_prober.go:28] interesting pod/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 14 19:43:14.948987 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:14.948629 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" podUID="f8b5510c-37c3-4935-a9fa-bf860b096000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 14 19:43:15.947797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:15.947755 2576 patch_prober.go:28] interesting pod/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 14 19:43:15.948051 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:15.947820 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" podUID="f8b5510c-37c3-4935-a9fa-bf860b096000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 14 19:43:16.948635 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:16.948560 2576 patch_prober.go:28] interesting pod/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 14 19:43:16.949040 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:16.948653 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" podUID="f8b5510c-37c3-4935-a9fa-bf860b096000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 14 19:43:17.947648 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:17.947576 2576 patch_prober.go:28] interesting pod/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 14 19:43:17.947866 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:17.947668 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" podUID="f8b5510c-37c3-4935-a9fa-bf860b096000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 14 19:43:18.509381 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:18.509343 2576 scope.go:117] "RemoveContainer" containerID="1a2ce54b2cd8403e80c574a3195dfa164fa8cac9c71b533bce677c1b50fcffb5" Jul 14 19:43:18.952059 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:18.952030 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:43:18.953032 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:18.953011 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:43:19.281233 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:19.281204 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n" event={"ID":"adfa7542-a27e-49af-9e0d-960a9057f3c9","Type":"ContainerStarted","Data":"ce68af6f0113dbc61fece03781f26ec06d3cf4a4de0a73a97103ed29ca52d359"} Jul 14 19:43:20.288308 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:20.288273 2576 generic.go:356] "Generic (PLEG): container finished" podID="adfa7542-a27e-49af-9e0d-960a9057f3c9" containerID="ce68af6f0113dbc61fece03781f26ec06d3cf4a4de0a73a97103ed29ca52d359" exitCode=0 Jul 14 19:43:20.288765 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:20.288363 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n" event={"ID":"adfa7542-a27e-49af-9e0d-960a9057f3c9","Type":"ContainerDied","Data":"ce68af6f0113dbc61fece03781f26ec06d3cf4a4de0a73a97103ed29ca52d359"} Jul 14 19:43:20.288765 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:20.288427 2576 scope.go:117] "RemoveContainer" containerID="1a2ce54b2cd8403e80c574a3195dfa164fa8cac9c71b533bce677c1b50fcffb5" Jul 14 19:43:20.312293 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:20.312257 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n"] Jul 14 19:43:20.430186 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:20.430158 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n" Jul 14 19:43:20.522479 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:20.522428 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfa7542-a27e-49af-9e0d-960a9057f3c9-maas-api-service-ca\") pod \"adfa7542-a27e-49af-9e0d-960a9057f3c9\" (UID: \"adfa7542-a27e-49af-9e0d-960a9057f3c9\") " Jul 14 19:43:20.522875 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:20.522844 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfa7542-a27e-49af-9e0d-960a9057f3c9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "adfa7542-a27e-49af-9e0d-960a9057f3c9" (UID: "adfa7542-a27e-49af-9e0d-960a9057f3c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:43:20.624197 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:20.624153 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfa7542-a27e-49af-9e0d-960a9057f3c9-maas-api-service-ca\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:43:21.294959 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:21.294924 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n" Jul 14 19:43:21.294959 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:21.294953 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n" event={"ID":"adfa7542-a27e-49af-9e0d-960a9057f3c9","Type":"ContainerDied","Data":"ec29835083b18a656d39db48f1fba98a4bd81d05d5c65b91091bca707983af90"} Jul 14 19:43:21.295515 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:21.295000 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec29835083b18a656d39db48f1fba98a4bd81d05d5c65b91091bca707983af90" Jul 14 19:43:21.326568 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:21.326534 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n"] Jul 14 19:43:21.334289 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:21.334258 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-xrr5n"] Jul 14 19:43:21.515642 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:21.515609 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adfa7542-a27e-49af-9e0d-960a9057f3c9" path="/var/lib/kubelet/pods/adfa7542-a27e-49af-9e0d-960a9057f3c9/volumes" Jul 14 19:43:52.192797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:52.192761 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-cgltx"] Jul 14 19:43:52.193387 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:52.193325 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adfa7542-a27e-49af-9e0d-960a9057f3c9" containerName="revoke-keys" Jul 14 19:43:52.193387 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:52.193347 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="adfa7542-a27e-49af-9e0d-960a9057f3c9" containerName="revoke-keys" Jul 14 19:43:52.193387 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:52.193378 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adfa7542-a27e-49af-9e0d-960a9057f3c9" containerName="revoke-keys" Jul 14 19:43:52.193387 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:52.193386 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="adfa7542-a27e-49af-9e0d-960a9057f3c9" containerName="revoke-keys" Jul 14 19:43:52.193654 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:52.193402 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adfa7542-a27e-49af-9e0d-960a9057f3c9" containerName="revoke-keys" Jul 14 19:43:52.193654 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:52.193410 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="adfa7542-a27e-49af-9e0d-960a9057f3c9" containerName="revoke-keys" Jul 14 19:43:52.193654 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:52.193510 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="adfa7542-a27e-49af-9e0d-960a9057f3c9" containerName="revoke-keys" Jul 14 19:43:52.193654 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:52.193521 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="adfa7542-a27e-49af-9e0d-960a9057f3c9" containerName="revoke-keys" Jul 14 19:43:52.196438 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:52.196417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-cgltx" Jul 14 19:43:52.200275 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:52.200255 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-br5f6\"" Jul 14 19:43:52.205660 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:52.205631 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-cgltx"] Jul 14 19:43:52.235693 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:52.235648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0025f4-d92d-4a08-b9ae-18686a8ae2b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9d072882-cgltx\" (UID: \"1a0025f4-d92d-4a08-b9ae-18686a8ae2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-cgltx" Jul 14 19:43:52.337084 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:52.337049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0025f4-d92d-4a08-b9ae-18686a8ae2b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9d072882-cgltx\" (UID: \"1a0025f4-d92d-4a08-b9ae-18686a8ae2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-cgltx" Jul 14 19:43:52.337860 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:52.337840 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0025f4-d92d-4a08-b9ae-18686a8ae2b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9d072882-cgltx\" (UID: \"1a0025f4-d92d-4a08-b9ae-18686a8ae2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-cgltx" Jul 14 19:43:52.509133 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:52.509035 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-cgltx" Jul 14 19:43:52.646183 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:52.646151 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-cgltx"] Jul 14 19:43:52.646870 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:43:52.646847 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0025f4_d92d_4a08_b9ae_18686a8ae2b6.slice/crio-315bf93b76b3c8d6cdefa1094c9e5e96e9fbbed85016ce499568d4212ef52365 WatchSource:0}: Error finding container 315bf93b76b3c8d6cdefa1094c9e5e96e9fbbed85016ce499568d4212ef52365: Status 404 returned error can't find the container with id 315bf93b76b3c8d6cdefa1094c9e5e96e9fbbed85016ce499568d4212ef52365 Jul 14 19:43:53.466735 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:53.466702 2576 generic.go:356] "Generic (PLEG): container finished" podID="1a0025f4-d92d-4a08-b9ae-18686a8ae2b6" containerID="58f075081173657903bbd6904e69f72a89f41a050302ceabc9f3c6916528d6e2" exitCode=0 Jul 14 19:43:53.467194 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:53.466791 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-cgltx" event={"ID":"1a0025f4-d92d-4a08-b9ae-18686a8ae2b6","Type":"ContainerDied","Data":"58f075081173657903bbd6904e69f72a89f41a050302ceabc9f3c6916528d6e2"} Jul 14 19:43:53.467194 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:53.466849 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-cgltx" event={"ID":"1a0025f4-d92d-4a08-b9ae-18686a8ae2b6","Type":"ContainerStarted","Data":"315bf93b76b3c8d6cdefa1094c9e5e96e9fbbed85016ce499568d4212ef52365"} Jul 14 19:43:54.608079 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:54.608051 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-cgltx" Jul 14 19:43:54.661206 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:54.661176 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0025f4-d92d-4a08-b9ae-18686a8ae2b6-maas-api-service-ca\") pod \"1a0025f4-d92d-4a08-b9ae-18686a8ae2b6\" (UID: \"1a0025f4-d92d-4a08-b9ae-18686a8ae2b6\") " Jul 14 19:43:54.661532 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:54.661509 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a0025f4-d92d-4a08-b9ae-18686a8ae2b6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1a0025f4-d92d-4a08-b9ae-18686a8ae2b6" (UID: "1a0025f4-d92d-4a08-b9ae-18686a8ae2b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:43:54.762598 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:54.762491 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0025f4-d92d-4a08-b9ae-18686a8ae2b6-maas-api-service-ca\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:43:55.478228 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:55.478196 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-cgltx" Jul 14 19:43:55.478406 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:55.478196 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d072882-cgltx" event={"ID":"1a0025f4-d92d-4a08-b9ae-18686a8ae2b6","Type":"ContainerDied","Data":"315bf93b76b3c8d6cdefa1094c9e5e96e9fbbed85016ce499568d4212ef52365"} Jul 14 19:43:55.478406 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:43:55.478323 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="315bf93b76b3c8d6cdefa1094c9e5e96e9fbbed85016ce499568d4212ef52365" Jul 14 19:44:17.470305 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:17.470269 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf"] Jul 14 19:44:17.470861 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:17.470539 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" podUID="f8b5510c-37c3-4935-a9fa-bf860b096000" containerName="istio-proxy" containerID="cri-o://a48622ad3949312beba9f1b15b86ce0f6cfb8d7f5d122dd2d3e622e952b5f76f" gracePeriod=30 Jul 14 19:44:18.954936 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:18.954893 2576 patch_prober.go:28] interesting pod/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 19:44:18.955341 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:18.954970 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" podUID="f8b5510c-37c3-4935-a9fa-bf860b096000" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 19:44:22.629206 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.629175 2576 generic.go:356] "Generic (PLEG): container finished" podID="f8b5510c-37c3-4935-a9fa-bf860b096000" containerID="a48622ad3949312beba9f1b15b86ce0f6cfb8d7f5d122dd2d3e622e952b5f76f" exitCode=0 Jul 14 19:44:22.629692 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.629228 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" event={"ID":"f8b5510c-37c3-4935-a9fa-bf860b096000","Type":"ContainerDied","Data":"a48622ad3949312beba9f1b15b86ce0f6cfb8d7f5d122dd2d3e622e952b5f76f"} Jul 14 19:44:22.629692 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.629275 2576 scope.go:117] "RemoveContainer" containerID="c572aebd79a88aa086015e547acd260ffd24448def4d662f7feee6254957ed56" Jul 14 19:44:22.721502 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.721474 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:44:22.836857 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.836825 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-workload-socket\") pod \"f8b5510c-37c3-4935-a9fa-bf860b096000\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " Jul 14 19:44:22.837025 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.836872 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-envoy\") pod \"f8b5510c-37c3-4935-a9fa-bf860b096000\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " Jul 14 19:44:22.837025 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.836915 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b5510c-37c3-4935-a9fa-bf860b096000-istiod-ca-cert\") pod \"f8b5510c-37c3-4935-a9fa-bf860b096000\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " Jul 14 19:44:22.837025 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.836938 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-token\") pod \"f8b5510c-37c3-4935-a9fa-bf860b096000\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " Jul 14 19:44:22.837025 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.836972 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-workload-certs\") pod \"f8b5510c-37c3-4935-a9fa-bf860b096000\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " Jul 14 19:44:22.837025 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.837013 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-credential-socket\") pod \"f8b5510c-37c3-4935-a9fa-bf860b096000\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " Jul 14 19:44:22.837294 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.837053 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-podinfo\") pod \"f8b5510c-37c3-4935-a9fa-bf860b096000\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " Jul 14 19:44:22.837294 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.837077 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzw5r\" (UniqueName: \"kubernetes.io/projected/f8b5510c-37c3-4935-a9fa-bf860b096000-kube-api-access-pzw5r\") pod \"f8b5510c-37c3-4935-a9fa-bf860b096000\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " Jul 14 19:44:22.837294 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.837109 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-data\") pod \"f8b5510c-37c3-4935-a9fa-bf860b096000\" (UID: \"f8b5510c-37c3-4935-a9fa-bf860b096000\") " Jul 14 19:44:22.837294 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.837145 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f8b5510c-37c3-4935-a9fa-bf860b096000" (UID: "f8b5510c-37c3-4935-a9fa-bf860b096000"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:44:22.837497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.837356 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f8b5510c-37c3-4935-a9fa-bf860b096000" (UID: "f8b5510c-37c3-4935-a9fa-bf860b096000"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:44:22.837497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.837389 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-workload-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:44:22.837497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.837408 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b5510c-37c3-4935-a9fa-bf860b096000-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f8b5510c-37c3-4935-a9fa-bf860b096000" (UID: "f8b5510c-37c3-4935-a9fa-bf860b096000"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:44:22.837677 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.837623 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f8b5510c-37c3-4935-a9fa-bf860b096000" (UID: "f8b5510c-37c3-4935-a9fa-bf860b096000"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:44:22.838255 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.838230 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-data" (OuterVolumeSpecName: "istio-data") pod "f8b5510c-37c3-4935-a9fa-bf860b096000" (UID: "f8b5510c-37c3-4935-a9fa-bf860b096000"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:44:22.839165 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.839145 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f8b5510c-37c3-4935-a9fa-bf860b096000" (UID: "f8b5510c-37c3-4935-a9fa-bf860b096000"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:44:22.839320 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.839304 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f8b5510c-37c3-4935-a9fa-bf860b096000" (UID: "f8b5510c-37c3-4935-a9fa-bf860b096000"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 19:44:22.839574 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.839549 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-token" (OuterVolumeSpecName: "istio-token") pod "f8b5510c-37c3-4935-a9fa-bf860b096000" (UID: "f8b5510c-37c3-4935-a9fa-bf860b096000"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:44:22.839574 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.839574 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b5510c-37c3-4935-a9fa-bf860b096000-kube-api-access-pzw5r" (OuterVolumeSpecName: "kube-api-access-pzw5r") pod "f8b5510c-37c3-4935-a9fa-bf860b096000" (UID: "f8b5510c-37c3-4935-a9fa-bf860b096000"). InnerVolumeSpecName "kube-api-access-pzw5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:44:22.938056 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.938007 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-envoy\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:44:22.938056 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.938052 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b5510c-37c3-4935-a9fa-bf860b096000-istiod-ca-cert\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:44:22.938056 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.938062 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-token\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:44:22.938056 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.938070 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-workload-certs\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:44:22.938334 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.938078 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-credential-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:44:22.938334 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.938087 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-podinfo\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:44:22.938334 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.938095 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzw5r\" (UniqueName: \"kubernetes.io/projected/f8b5510c-37c3-4935-a9fa-bf860b096000-kube-api-access-pzw5r\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:44:22.938334 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:22.938103 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b5510c-37c3-4935-a9fa-bf860b096000-istio-data\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:44:23.635846 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:23.635817 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" Jul 14 19:44:23.635846 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:23.635827 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf" event={"ID":"f8b5510c-37c3-4935-a9fa-bf860b096000","Type":"ContainerDied","Data":"24be63ed63cab75f0e54d8d2133065a7589611c1b3dc87b4b8791526def1a010"} Jul 14 19:44:23.636343 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:23.635880 2576 scope.go:117] "RemoveContainer" containerID="a48622ad3949312beba9f1b15b86ce0f6cfb8d7f5d122dd2d3e622e952b5f76f" Jul 14 19:44:23.657109 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:23.657072 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf"] Jul 14 19:44:23.660638 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:23.660607 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-9d072882-openshift-default-5fdf49dbc-8dcdf"] Jul 14 19:44:25.516080 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:44:25.516043 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8b5510c-37c3-4935-a9fa-bf860b096000" path="/var/lib/kubelet/pods/f8b5510c-37c3-4935-a9fa-bf860b096000/volumes" Jul 14 19:45:47.558412 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:47.558379 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:45:47.566293 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:47.566268 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:45:56.950433 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:56.950398 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq"] Jul 14 19:45:56.951540 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:56.951514 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8b5510c-37c3-4935-a9fa-bf860b096000" containerName="istio-proxy" Jul 14 19:45:56.951640 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:56.951542 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b5510c-37c3-4935-a9fa-bf860b096000" containerName="istio-proxy" Jul 14 19:45:56.951640 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:56.951555 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a0025f4-d92d-4a08-b9ae-18686a8ae2b6" containerName="revoke-keys" Jul 14 19:45:56.951640 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:56.951563 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a0025f4-d92d-4a08-b9ae-18686a8ae2b6" containerName="revoke-keys" Jul 14 19:45:56.951640 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:56.951576 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8b5510c-37c3-4935-a9fa-bf860b096000" containerName="istio-proxy" Jul 14 19:45:56.951640 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:56.951609 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b5510c-37c3-4935-a9fa-bf860b096000" containerName="istio-proxy" Jul 14 19:45:56.951798 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:56.951723 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="adfa7542-a27e-49af-9e0d-960a9057f3c9" containerName="revoke-keys" Jul 14 19:45:56.951798 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:56.951734 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8b5510c-37c3-4935-a9fa-bf860b096000" containerName="istio-proxy" Jul 14 19:45:56.951798 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:56.951744 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8b5510c-37c3-4935-a9fa-bf860b096000" containerName="istio-proxy" Jul 14 19:45:56.951798 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:56.951755 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a0025f4-d92d-4a08-b9ae-18686a8ae2b6" containerName="revoke-keys" Jul 14 19:45:56.955259 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:56.955242 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:56.958076 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:56.958054 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9b84c5bc-openshift-default-dockercfg-pbrbg\"" Jul 14 19:45:56.967493 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:56.967471 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq"] Jul 14 19:45:57.034220 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.034190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-envoy\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.034396 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.034237 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-workload-socket\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.034396 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.034287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-workload-certs\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.034396 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.034345 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istiod-ca-cert\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.034396 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.034377 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-credential-socket\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.034658 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.034450 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-token\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.034658 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.034495 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-podinfo\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.034658 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.034525 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtz7k\" (UniqueName: \"kubernetes.io/projected/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-kube-api-access-gtz7k\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.034658 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.034623 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-data\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.135549 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.135504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-workload-socket\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.135549 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.135551 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-workload-certs\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.135825 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.135678 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istiod-ca-cert\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.135825 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.135726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-credential-socket\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.135825 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.135789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-token\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.135995 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.135835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-podinfo\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.135995 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.135872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtz7k\" (UniqueName: \"kubernetes.io/projected/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-kube-api-access-gtz7k\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.135995 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.135944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-data\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.135995 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.135979 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-workload-certs\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.136197 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.135994 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-envoy\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.136197 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.136041 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-workload-socket\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.136197 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.136089 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-credential-socket\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.136370 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.136330 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-data\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.136410 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.136377 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istiod-ca-cert\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.138172 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.138143 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-envoy\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.138446 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.138429 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-podinfo\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.145142 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.145116 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-token\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.145253 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.145238 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtz7k\" (UniqueName: \"kubernetes.io/projected/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-kube-api-access-gtz7k\") pod \"e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.266528 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.266428 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:57.416655 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.416625 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq"] Jul 14 19:45:57.418708 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:45:57.418683 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbba22a_dab6_4f8d_9fc0_3736949f40d7.slice/crio-d4b587e0cfa1e194100529c4ed971b983a569bae4ba947e37fa18444f5bd0ebf WatchSource:0}: Error finding container d4b587e0cfa1e194100529c4ed971b983a569bae4ba947e37fa18444f5bd0ebf: Status 404 returned error can't find the container with id d4b587e0cfa1e194100529c4ed971b983a569bae4ba947e37fa18444f5bd0ebf Jul 14 19:45:57.420935 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.420902 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:45:57.421014 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.420964 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:45:57.421014 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:57.420996 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:45:58.115334 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:58.115298 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" event={"ID":"3fbba22a-dab6-4f8d-9fc0-3736949f40d7","Type":"ContainerStarted","Data":"071d688acf037b776b223b4b09ea7a7606d6665f2d4b701b38f70f521ff9cfb6"} Jul 14 19:45:58.115758 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:58.115377 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" event={"ID":"3fbba22a-dab6-4f8d-9fc0-3736949f40d7","Type":"ContainerStarted","Data":"d4b587e0cfa1e194100529c4ed971b983a569bae4ba947e37fa18444f5bd0ebf"} Jul 14 19:45:58.155097 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:58.155038 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" podStartSLOduration=2.15501966 podStartE2EDuration="2.15501966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:45:58.153088717 +0000 UTC m=+2111.266338588" watchObservedRunningTime="2026-07-14 19:45:58.15501966 +0000 UTC m=+2111.268269532" Jul 14 19:45:58.266960 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:58.266929 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:58.276127 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:58.276098 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:59.120550 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:59.120521 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:45:59.121573 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:45:59.121552 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:46:12.401664 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:12.401628 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt"] Jul 14 19:46:12.408535 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:12.408511 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt" Jul 14 19:46:12.413050 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:12.413026 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-br5f6\"" Jul 14 19:46:12.418595 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:12.418552 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt"] Jul 14 19:46:12.506830 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:12.506794 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3787b1bc-7c59-42b7-8f60-8b1dac53a2b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt\" (UID: \"3787b1bc-7c59-42b7-8f60-8b1dac53a2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt" Jul 14 19:46:12.607978 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:12.607934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3787b1bc-7c59-42b7-8f60-8b1dac53a2b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt\" (UID: \"3787b1bc-7c59-42b7-8f60-8b1dac53a2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt" Jul 14 19:46:12.608548 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:12.608528 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3787b1bc-7c59-42b7-8f60-8b1dac53a2b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt\" (UID: \"3787b1bc-7c59-42b7-8f60-8b1dac53a2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt" Jul 14 19:46:12.722048 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:12.721966 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt" Jul 14 19:46:12.867653 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:12.867627 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt"] Jul 14 19:46:12.870249 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:46:12.870219 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3787b1bc_7c59_42b7_8f60_8b1dac53a2b7.slice/crio-54d0b7f4ecad6f06ad9738da9d8f10690ff8d04b68829ca2525fdb10fa882111 WatchSource:0}: Error finding container 54d0b7f4ecad6f06ad9738da9d8f10690ff8d04b68829ca2525fdb10fa882111: Status 404 returned error can't find the container with id 54d0b7f4ecad6f06ad9738da9d8f10690ff8d04b68829ca2525fdb10fa882111 Jul 14 19:46:13.199566 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:13.199533 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt" event={"ID":"3787b1bc-7c59-42b7-8f60-8b1dac53a2b7","Type":"ContainerStarted","Data":"17d2055117a9626e833013c20427acab4b4d8da8d802785905108689ecbcb42f"} Jul 14 19:46:13.199738 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:13.199613 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt" event={"ID":"3787b1bc-7c59-42b7-8f60-8b1dac53a2b7","Type":"ContainerStarted","Data":"54d0b7f4ecad6f06ad9738da9d8f10690ff8d04b68829ca2525fdb10fa882111"} Jul 14 19:46:13.216766 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:13.216719 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt" podStartSLOduration=1.216704228 podStartE2EDuration="1.216704228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:13.214949156 +0000 UTC m=+2126.328199038" watchObservedRunningTime="2026-07-14 19:46:13.216704228 +0000 UTC m=+2126.329954098" Jul 14 19:46:14.122516 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:14.122482 2576 patch_prober.go:28] interesting pod/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 14 19:46:14.122951 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:14.122536 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" podUID="3fbba22a-dab6-4f8d-9fc0-3736949f40d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 14 19:46:14.206095 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:14.206063 2576 generic.go:356] "Generic (PLEG): container finished" podID="3787b1bc-7c59-42b7-8f60-8b1dac53a2b7" containerID="17d2055117a9626e833013c20427acab4b4d8da8d802785905108689ecbcb42f" exitCode=0 Jul 14 19:46:14.206253 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:14.206120 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt" event={"ID":"3787b1bc-7c59-42b7-8f60-8b1dac53a2b7","Type":"ContainerDied","Data":"17d2055117a9626e833013c20427acab4b4d8da8d802785905108689ecbcb42f"} Jul 14 19:46:14.207703 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:14.207681 2576 generic.go:356] "Generic (PLEG): container finished" podID="3fbba22a-dab6-4f8d-9fc0-3736949f40d7" containerID="071d688acf037b776b223b4b09ea7a7606d6665f2d4b701b38f70f521ff9cfb6" exitCode=0 Jul 14 19:46:14.207828 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:14.207714 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" event={"ID":"3fbba22a-dab6-4f8d-9fc0-3736949f40d7","Type":"ContainerDied","Data":"071d688acf037b776b223b4b09ea7a7606d6665f2d4b701b38f70f521ff9cfb6"} Jul 14 19:46:14.208097 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:14.208085 2576 scope.go:117] "RemoveContainer" containerID="071d688acf037b776b223b4b09ea7a7606d6665f2d4b701b38f70f521ff9cfb6" Jul 14 19:46:14.209085 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:14.209061 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:46:14.209134 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:14.209125 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:46:14.209177 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:14.209153 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:46:15.214783 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:15.214746 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" event={"ID":"3fbba22a-dab6-4f8d-9fc0-3736949f40d7","Type":"ContainerStarted","Data":"ac71c8666d824216dbfa3a02b752f6c59494de2b7d6e7c471f13e9a237e827b0"} Jul 14 19:46:15.266856 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:15.266666 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:46:15.267041 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:15.267017 2576 patch_prober.go:28] interesting pod/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 14 19:46:15.267127 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:15.267070 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" podUID="3fbba22a-dab6-4f8d-9fc0-3736949f40d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 14 19:46:15.358687 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:15.358662 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt" Jul 14 19:46:15.436231 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:15.436196 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3787b1bc-7c59-42b7-8f60-8b1dac53a2b7-maas-api-service-ca\") pod \"3787b1bc-7c59-42b7-8f60-8b1dac53a2b7\" (UID: \"3787b1bc-7c59-42b7-8f60-8b1dac53a2b7\") " Jul 14 19:46:15.436570 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:15.436545 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3787b1bc-7c59-42b7-8f60-8b1dac53a2b7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3787b1bc-7c59-42b7-8f60-8b1dac53a2b7" (UID: "3787b1bc-7c59-42b7-8f60-8b1dac53a2b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:46:15.537479 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:15.537447 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3787b1bc-7c59-42b7-8f60-8b1dac53a2b7-maas-api-service-ca\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:46:16.221137 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:16.221103 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt" Jul 14 19:46:16.221561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:16.221134 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt" event={"ID":"3787b1bc-7c59-42b7-8f60-8b1dac53a2b7","Type":"ContainerDied","Data":"54d0b7f4ecad6f06ad9738da9d8f10690ff8d04b68829ca2525fdb10fa882111"} Jul 14 19:46:16.221561 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:16.221179 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54d0b7f4ecad6f06ad9738da9d8f10690ff8d04b68829ca2525fdb10fa882111" Jul 14 19:46:16.267540 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:16.267497 2576 patch_prober.go:28] interesting pod/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 14 19:46:16.267734 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:16.267562 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" podUID="3fbba22a-dab6-4f8d-9fc0-3736949f40d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 14 19:46:17.266824 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:17.266785 2576 patch_prober.go:28] interesting pod/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 14 19:46:17.267326 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:17.266863 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" podUID="3fbba22a-dab6-4f8d-9fc0-3736949f40d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 14 19:46:18.267817 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:18.267779 2576 patch_prober.go:28] interesting pod/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 14 19:46:18.268254 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:18.267848 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" podUID="3fbba22a-dab6-4f8d-9fc0-3736949f40d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 14 19:46:19.271153 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:19.271109 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:46:19.271676 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:19.271444 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:46:19.272117 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:19.272101 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:46:38.257983 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:38.257945 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq"] Jul 14 19:46:38.258595 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:38.258211 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" podUID="3fbba22a-dab6-4f8d-9fc0-3736949f40d7" containerName="istio-proxy" containerID="cri-o://ac71c8666d824216dbfa3a02b752f6c59494de2b7d6e7c471f13e9a237e827b0" gracePeriod=30 Jul 14 19:46:43.525344 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.525271 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:46:43.613397 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.613361 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-workload-certs\") pod \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " Jul 14 19:46:43.613397 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.613401 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-podinfo\") pod \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " Jul 14 19:46:43.613683 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.613432 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istiod-ca-cert\") pod \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " Jul 14 19:46:43.613683 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.613473 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-workload-socket\") pod \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " Jul 14 19:46:43.613683 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.613510 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-envoy\") pod \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " Jul 14 19:46:43.613683 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.613560 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtz7k\" (UniqueName: \"kubernetes.io/projected/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-kube-api-access-gtz7k\") pod \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " Jul 14 19:46:43.613683 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.613576 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-data\") pod \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " Jul 14 19:46:43.613683 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.613639 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-credential-socket\") pod \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " Jul 14 19:46:43.613683 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.613685 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-token\") pod \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\" (UID: \"3fbba22a-dab6-4f8d-9fc0-3736949f40d7\") " Jul 14 19:46:43.614111 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.613855 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3fbba22a-dab6-4f8d-9fc0-3736949f40d7" (UID: "3fbba22a-dab6-4f8d-9fc0-3736949f40d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:46:43.614111 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.614055 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3fbba22a-dab6-4f8d-9fc0-3736949f40d7" (UID: "3fbba22a-dab6-4f8d-9fc0-3736949f40d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:46:43.614111 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.614079 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3fbba22a-dab6-4f8d-9fc0-3736949f40d7" (UID: "3fbba22a-dab6-4f8d-9fc0-3736949f40d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:46:43.614246 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.614173 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3fbba22a-dab6-4f8d-9fc0-3736949f40d7" (UID: "3fbba22a-dab6-4f8d-9fc0-3736949f40d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:46:43.614646 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.614622 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-data" (OuterVolumeSpecName: "istio-data") pod "3fbba22a-dab6-4f8d-9fc0-3736949f40d7" (UID: "3fbba22a-dab6-4f8d-9fc0-3736949f40d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:46:43.615677 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.615651 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3fbba22a-dab6-4f8d-9fc0-3736949f40d7" (UID: "3fbba22a-dab6-4f8d-9fc0-3736949f40d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:46:43.615881 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.615866 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-token" (OuterVolumeSpecName: "istio-token") pod "3fbba22a-dab6-4f8d-9fc0-3736949f40d7" (UID: "3fbba22a-dab6-4f8d-9fc0-3736949f40d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:46:43.616083 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.616058 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3fbba22a-dab6-4f8d-9fc0-3736949f40d7" (UID: "3fbba22a-dab6-4f8d-9fc0-3736949f40d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 19:46:43.616217 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.616195 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-kube-api-access-gtz7k" (OuterVolumeSpecName: "kube-api-access-gtz7k") pod "3fbba22a-dab6-4f8d-9fc0-3736949f40d7" (UID: "3fbba22a-dab6-4f8d-9fc0-3736949f40d7"). InnerVolumeSpecName "kube-api-access-gtz7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:46:43.714563 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.714525 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istiod-ca-cert\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:46:43.714563 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.714557 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-workload-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:46:43.714563 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.714566 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-envoy\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:46:43.714563 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.714575 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtz7k\" (UniqueName: \"kubernetes.io/projected/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-kube-api-access-gtz7k\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:46:43.714863 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.714612 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-data\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:46:43.714863 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.714620 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-credential-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:46:43.714863 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.714628 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-token\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:46:43.714863 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.714636 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-workload-certs\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:46:43.714863 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:43.714645 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fbba22a-dab6-4f8d-9fc0-3736949f40d7-istio-podinfo\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:46:44.371677 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:44.371638 2576 generic.go:356] "Generic (PLEG): container finished" podID="3fbba22a-dab6-4f8d-9fc0-3736949f40d7" containerID="ac71c8666d824216dbfa3a02b752f6c59494de2b7d6e7c471f13e9a237e827b0" exitCode=0 Jul 14 19:46:44.371881 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:44.371692 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" Jul 14 19:46:44.371881 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:44.371724 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" event={"ID":"3fbba22a-dab6-4f8d-9fc0-3736949f40d7","Type":"ContainerDied","Data":"ac71c8666d824216dbfa3a02b752f6c59494de2b7d6e7c471f13e9a237e827b0"} Jul 14 19:46:44.371881 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:44.371808 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq" event={"ID":"3fbba22a-dab6-4f8d-9fc0-3736949f40d7","Type":"ContainerDied","Data":"d4b587e0cfa1e194100529c4ed971b983a569bae4ba947e37fa18444f5bd0ebf"} Jul 14 19:46:44.371881 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:44.371831 2576 scope.go:117] "RemoveContainer" containerID="ac71c8666d824216dbfa3a02b752f6c59494de2b7d6e7c471f13e9a237e827b0" Jul 14 19:46:44.381171 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:44.381042 2576 scope.go:117] "RemoveContainer" containerID="071d688acf037b776b223b4b09ea7a7606d6665f2d4b701b38f70f521ff9cfb6" Jul 14 19:46:44.389393 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:44.389374 2576 scope.go:117] "RemoveContainer" containerID="ac71c8666d824216dbfa3a02b752f6c59494de2b7d6e7c471f13e9a237e827b0" Jul 14 19:46:44.389683 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:46:44.389658 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac71c8666d824216dbfa3a02b752f6c59494de2b7d6e7c471f13e9a237e827b0\": container with ID starting with ac71c8666d824216dbfa3a02b752f6c59494de2b7d6e7c471f13e9a237e827b0 not found: ID does not exist" containerID="ac71c8666d824216dbfa3a02b752f6c59494de2b7d6e7c471f13e9a237e827b0" Jul 14 19:46:44.389773 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:44.389696 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac71c8666d824216dbfa3a02b752f6c59494de2b7d6e7c471f13e9a237e827b0"} err="failed to get container status \"ac71c8666d824216dbfa3a02b752f6c59494de2b7d6e7c471f13e9a237e827b0\": rpc error: code = NotFound desc = could not find container \"ac71c8666d824216dbfa3a02b752f6c59494de2b7d6e7c471f13e9a237e827b0\": container with ID starting with ac71c8666d824216dbfa3a02b752f6c59494de2b7d6e7c471f13e9a237e827b0 not found: ID does not exist" Jul 14 19:46:44.389773 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:44.389721 2576 scope.go:117] "RemoveContainer" containerID="071d688acf037b776b223b4b09ea7a7606d6665f2d4b701b38f70f521ff9cfb6" Jul 14 19:46:44.389994 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:46:44.389978 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"071d688acf037b776b223b4b09ea7a7606d6665f2d4b701b38f70f521ff9cfb6\": container with ID starting with 071d688acf037b776b223b4b09ea7a7606d6665f2d4b701b38f70f521ff9cfb6 not found: ID does not exist" containerID="071d688acf037b776b223b4b09ea7a7606d6665f2d4b701b38f70f521ff9cfb6" Jul 14 19:46:44.390052 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:44.390003 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"071d688acf037b776b223b4b09ea7a7606d6665f2d4b701b38f70f521ff9cfb6"} err="failed to get container status \"071d688acf037b776b223b4b09ea7a7606d6665f2d4b701b38f70f521ff9cfb6\": rpc error: code = NotFound desc = could not find container \"071d688acf037b776b223b4b09ea7a7606d6665f2d4b701b38f70f521ff9cfb6\": container with ID starting with 071d688acf037b776b223b4b09ea7a7606d6665f2d4b701b38f70f521ff9cfb6 not found: ID does not exist" Jul 14 19:46:44.393639 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:44.393617 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq"] Jul 14 19:46:44.399272 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:44.399244 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9b84c5bc-openshift-default-5b97ddd95f-8jvvq"] Jul 14 19:46:45.515685 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:45.515650 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fbba22a-dab6-4f8d-9fc0-3736949f40d7" path="/var/lib/kubelet/pods/3fbba22a-dab6-4f8d-9fc0-3736949f40d7/volumes" Jul 14 19:46:59.089562 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.089528 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf"] Jul 14 19:46:59.090034 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.089993 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3787b1bc-7c59-42b7-8f60-8b1dac53a2b7" containerName="revoke-keys" Jul 14 19:46:59.090034 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.090007 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3787b1bc-7c59-42b7-8f60-8b1dac53a2b7" containerName="revoke-keys" Jul 14 19:46:59.090034 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.090019 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fbba22a-dab6-4f8d-9fc0-3736949f40d7" containerName="istio-proxy" Jul 14 19:46:59.090034 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.090024 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fbba22a-dab6-4f8d-9fc0-3736949f40d7" containerName="istio-proxy" Jul 14 19:46:59.090188 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.090121 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fbba22a-dab6-4f8d-9fc0-3736949f40d7" containerName="istio-proxy" Jul 14 19:46:59.090188 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.090133 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fbba22a-dab6-4f8d-9fc0-3736949f40d7" containerName="istio-proxy" Jul 14 19:46:59.090188 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.090147 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3787b1bc-7c59-42b7-8f60-8b1dac53a2b7" containerName="revoke-keys" Jul 14 19:46:59.093706 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.093683 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.096758 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.096734 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ea441188-openshift-default-dockercfg-cr5fv\"" Jul 14 19:46:59.107995 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.107968 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf"] Jul 14 19:46:59.158072 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.158034 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-token\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.158265 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.158092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istiod-ca-cert\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.158265 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.158118 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-data\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.158265 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.158168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-credential-socket\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.158265 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.158184 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl9dc\" (UniqueName: \"kubernetes.io/projected/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-kube-api-access-wl9dc\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.158265 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.158248 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-workload-socket\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.158434 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.158268 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-podinfo\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.158434 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.158288 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-envoy\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.158434 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.158337 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-workload-certs\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.259544 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.259509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-workload-socket\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.259544 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.259551 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-podinfo\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.259850 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.259569 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-envoy\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.259850 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.259611 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-workload-certs\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.259850 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.259664 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-token\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.259850 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.259703 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istiod-ca-cert\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.259850 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.259721 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-data\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.259850 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.259753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-credential-socket\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.259850 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.259776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl9dc\" (UniqueName: \"kubernetes.io/projected/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-kube-api-access-wl9dc\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.260231 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.260003 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-workload-socket\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.260231 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.260032 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-workload-certs\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.260231 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.260142 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-data\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.260231 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.260223 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-credential-socket\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.260834 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.260809 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istiod-ca-cert\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.262263 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.262227 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-envoy\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.262429 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.262411 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-podinfo\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.279154 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.279124 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-token\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.280443 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.280420 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl9dc\" (UniqueName: \"kubernetes.io/projected/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-kube-api-access-wl9dc\") pod \"e2e-mt-ea441188-openshift-default-d997bf747-f8prf\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.405037 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.404944 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:46:59.553937 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.553907 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf"] Jul 14 19:46:59.555473 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:46:59.555450 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1370edf_d7bf_42c1_b14a_1b0f4617fe27.slice/crio-151401012a2e16e17478efa0d10ee40b7f203be00f734c1d7dd69227115686d7 WatchSource:0}: Error finding container 151401012a2e16e17478efa0d10ee40b7f203be00f734c1d7dd69227115686d7: Status 404 returned error can't find the container with id 151401012a2e16e17478efa0d10ee40b7f203be00f734c1d7dd69227115686d7 Jul 14 19:46:59.557682 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.557648 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:46:59.557767 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.557715 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:46:59.557767 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:46:59.557743 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:47:00.453794 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:00.453762 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" event={"ID":"e1370edf-d7bf-42c1-b14a-1b0f4617fe27","Type":"ContainerStarted","Data":"9713f6b7fe181db309061615e3c6f0378438cc3c47e81607d61ae977cd7ed04a"} Jul 14 19:47:00.454191 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:00.453816 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" event={"ID":"e1370edf-d7bf-42c1-b14a-1b0f4617fe27","Type":"ContainerStarted","Data":"151401012a2e16e17478efa0d10ee40b7f203be00f734c1d7dd69227115686d7"} Jul 14 19:47:00.477906 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:00.477848 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" podStartSLOduration=1.477831267 podStartE2EDuration="1.477831267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:00.476526304 +0000 UTC m=+2173.589776177" watchObservedRunningTime="2026-07-14 19:47:00.477831267 +0000 UTC m=+2173.591081134" Jul 14 19:47:01.406093 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:01.406049 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:47:01.410814 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:01.410789 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:47:01.459184 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:01.459159 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:47:01.460267 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:01.460247 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:47:15.537442 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:15.537407 2576 generic.go:356] "Generic (PLEG): container finished" podID="e1370edf-d7bf-42c1-b14a-1b0f4617fe27" containerID="9713f6b7fe181db309061615e3c6f0378438cc3c47e81607d61ae977cd7ed04a" exitCode=0 Jul 14 19:47:15.537875 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:15.537475 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" event={"ID":"e1370edf-d7bf-42c1-b14a-1b0f4617fe27","Type":"ContainerDied","Data":"9713f6b7fe181db309061615e3c6f0378438cc3c47e81607d61ae977cd7ed04a"} Jul 14 19:47:15.537925 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:15.537904 2576 scope.go:117] "RemoveContainer" containerID="9713f6b7fe181db309061615e3c6f0378438cc3c47e81607d61ae977cd7ed04a" Jul 14 19:47:15.539059 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:15.539031 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:47:15.539141 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:15.539098 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:47:15.539188 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:15.539157 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:47:16.459870 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:16.459829 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:47:16.544163 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:16.544128 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" event={"ID":"e1370edf-d7bf-42c1-b14a-1b0f4617fe27","Type":"ContainerStarted","Data":"056b98ec2f43d27f807c1a0a8e7320c1096458a428980551f4757d451404f015"} Jul 14 19:47:17.405986 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:17.405951 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:47:17.407316 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:17.407286 2576 patch_prober.go:28] interesting pod/e2e-mt-ea441188-openshift-default-d997bf747-f8prf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 14 19:47:17.407462 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:17.407344 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" podUID="e1370edf-d7bf-42c1-b14a-1b0f4617fe27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 14 19:47:18.406282 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:18.406241 2576 patch_prober.go:28] interesting pod/e2e-mt-ea441188-openshift-default-d997bf747-f8prf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 14 19:47:18.406701 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:18.406317 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" podUID="e1370edf-d7bf-42c1-b14a-1b0f4617fe27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 14 19:47:19.405721 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:19.405677 2576 patch_prober.go:28] interesting pod/e2e-mt-ea441188-openshift-default-d997bf747-f8prf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 14 19:47:19.405911 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:19.405745 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" podUID="e1370edf-d7bf-42c1-b14a-1b0f4617fe27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 14 19:47:20.406434 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:20.406394 2576 patch_prober.go:28] interesting pod/e2e-mt-ea441188-openshift-default-d997bf747-f8prf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 14 19:47:20.406960 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:20.406460 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" podUID="e1370edf-d7bf-42c1-b14a-1b0f4617fe27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 14 19:47:21.410018 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:21.409986 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:47:21.410405 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:21.410375 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:47:21.411178 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:21.411157 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:47:40.980403 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:40.980363 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea441188-qwb2x"] Jul 14 19:47:40.980942 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:40.980925 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fbba22a-dab6-4f8d-9fc0-3736949f40d7" containerName="istio-proxy" Jul 14 19:47:40.980988 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:40.980945 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fbba22a-dab6-4f8d-9fc0-3736949f40d7" containerName="istio-proxy" Jul 14 19:47:40.983734 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:40.983717 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea441188-qwb2x" Jul 14 19:47:40.986685 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:40.986664 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-br5f6\"" Jul 14 19:47:40.991512 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:40.991486 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea441188-qwb2x"] Jul 14 19:47:41.085774 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:41.085735 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f83598f-2c57-46b1-abe7-87aef32d89a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ea441188-qwb2x\" (UID: \"4f83598f-2c57-46b1-abe7-87aef32d89a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea441188-qwb2x" Jul 14 19:47:41.186717 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:41.186670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f83598f-2c57-46b1-abe7-87aef32d89a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ea441188-qwb2x\" (UID: \"4f83598f-2c57-46b1-abe7-87aef32d89a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea441188-qwb2x" Jul 14 19:47:41.187288 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:41.187270 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f83598f-2c57-46b1-abe7-87aef32d89a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ea441188-qwb2x\" (UID: \"4f83598f-2c57-46b1-abe7-87aef32d89a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea441188-qwb2x" Jul 14 19:47:41.296003 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:41.295922 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea441188-qwb2x" Jul 14 19:47:41.437282 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:41.437237 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea441188-qwb2x"] Jul 14 19:47:41.439727 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:47:41.439701 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f83598f_2c57_46b1_abe7_87aef32d89a6.slice/crio-1e4ca642a1652648786a0295acbbb32bbec3f22cb0c732a76abad6c621a003ba WatchSource:0}: Error finding container 1e4ca642a1652648786a0295acbbb32bbec3f22cb0c732a76abad6c621a003ba: Status 404 returned error can't find the container with id 1e4ca642a1652648786a0295acbbb32bbec3f22cb0c732a76abad6c621a003ba Jul 14 19:47:41.678619 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:41.678567 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea441188-qwb2x" event={"ID":"4f83598f-2c57-46b1-abe7-87aef32d89a6","Type":"ContainerStarted","Data":"1e4ca642a1652648786a0295acbbb32bbec3f22cb0c732a76abad6c621a003ba"} Jul 14 19:47:42.685988 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:42.685953 2576 generic.go:356] "Generic (PLEG): container finished" podID="4f83598f-2c57-46b1-abe7-87aef32d89a6" containerID="e9d203e7c8883d4569cf45765fdc8dd5c2275b3724c68c89ecaa223fc87067cf" exitCode=0 Jul 14 19:47:42.686381 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:42.685991 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea441188-qwb2x" event={"ID":"4f83598f-2c57-46b1-abe7-87aef32d89a6","Type":"ContainerDied","Data":"e9d203e7c8883d4569cf45765fdc8dd5c2275b3724c68c89ecaa223fc87067cf"} Jul 14 19:47:43.831945 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:43.831920 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea441188-qwb2x" Jul 14 19:47:43.915110 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:43.915081 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f83598f-2c57-46b1-abe7-87aef32d89a6-maas-api-service-ca\") pod \"4f83598f-2c57-46b1-abe7-87aef32d89a6\" (UID: \"4f83598f-2c57-46b1-abe7-87aef32d89a6\") " Jul 14 19:47:43.915417 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:43.915393 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f83598f-2c57-46b1-abe7-87aef32d89a6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4f83598f-2c57-46b1-abe7-87aef32d89a6" (UID: "4f83598f-2c57-46b1-abe7-87aef32d89a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:47:44.016727 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:44.016635 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f83598f-2c57-46b1-abe7-87aef32d89a6-maas-api-service-ca\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:47:44.698653 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:44.698622 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea441188-qwb2x" Jul 14 19:47:44.698864 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:44.698653 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ea441188-qwb2x" event={"ID":"4f83598f-2c57-46b1-abe7-87aef32d89a6","Type":"ContainerDied","Data":"1e4ca642a1652648786a0295acbbb32bbec3f22cb0c732a76abad6c621a003ba"} Jul 14 19:47:44.698864 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:47:44.698702 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e4ca642a1652648786a0295acbbb32bbec3f22cb0c732a76abad6c621a003ba" Jul 14 19:49:08.559965 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:08.559931 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf"] Jul 14 19:49:08.560454 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:08.560153 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" podUID="e1370edf-d7bf-42c1-b14a-1b0f4617fe27" containerName="istio-proxy" containerID="cri-o://056b98ec2f43d27f807c1a0a8e7320c1096458a428980551f4757d451404f015" gracePeriod=30 Jul 14 19:49:13.812895 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.812865 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:49:13.995483 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.995394 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-workload-certs\") pod \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " Jul 14 19:49:13.995483 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.995458 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istiod-ca-cert\") pod \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " Jul 14 19:49:13.995483 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.995482 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-token\") pod \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " Jul 14 19:49:13.995797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.995512 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-envoy\") pod \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " Jul 14 19:49:13.995797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.995534 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-workload-socket\") pod \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " Jul 14 19:49:13.995797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.995572 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-data\") pod \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " Jul 14 19:49:13.995797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.995647 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-podinfo\") pod \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " Jul 14 19:49:13.995797 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.995726 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e1370edf-d7bf-42c1-b14a-1b0f4617fe27" (UID: "e1370edf-d7bf-42c1-b14a-1b0f4617fe27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:49:13.996054 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.995853 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-credential-socket\") pod \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " Jul 14 19:49:13.996054 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.995892 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl9dc\" (UniqueName: \"kubernetes.io/projected/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-kube-api-access-wl9dc\") pod \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\" (UID: \"e1370edf-d7bf-42c1-b14a-1b0f4617fe27\") " Jul 14 19:49:13.996054 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.995915 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e1370edf-d7bf-42c1-b14a-1b0f4617fe27" (UID: "e1370edf-d7bf-42c1-b14a-1b0f4617fe27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:49:13.996054 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.995953 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e1370edf-d7bf-42c1-b14a-1b0f4617fe27" (UID: "e1370edf-d7bf-42c1-b14a-1b0f4617fe27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:49:13.996277 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.996107 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e1370edf-d7bf-42c1-b14a-1b0f4617fe27" (UID: "e1370edf-d7bf-42c1-b14a-1b0f4617fe27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:49:13.996277 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.996257 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-credential-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:49:13.996277 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.996276 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-workload-certs\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:49:13.996427 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.996290 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istiod-ca-cert\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:49:13.996427 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.996302 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-workload-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:49:13.996710 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.996679 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-data" (OuterVolumeSpecName: "istio-data") pod "e1370edf-d7bf-42c1-b14a-1b0f4617fe27" (UID: "e1370edf-d7bf-42c1-b14a-1b0f4617fe27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:49:13.997839 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.997807 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-token" (OuterVolumeSpecName: "istio-token") pod "e1370edf-d7bf-42c1-b14a-1b0f4617fe27" (UID: "e1370edf-d7bf-42c1-b14a-1b0f4617fe27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:49:13.997954 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.997921 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-kube-api-access-wl9dc" (OuterVolumeSpecName: "kube-api-access-wl9dc") pod "e1370edf-d7bf-42c1-b14a-1b0f4617fe27" (UID: "e1370edf-d7bf-42c1-b14a-1b0f4617fe27"). InnerVolumeSpecName "kube-api-access-wl9dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:49:13.998092 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.998057 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e1370edf-d7bf-42c1-b14a-1b0f4617fe27" (UID: "e1370edf-d7bf-42c1-b14a-1b0f4617fe27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 19:49:13.998092 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:13.998076 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e1370edf-d7bf-42c1-b14a-1b0f4617fe27" (UID: "e1370edf-d7bf-42c1-b14a-1b0f4617fe27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:49:14.097790 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:14.097754 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-data\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:49:14.097790 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:14.097782 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-podinfo\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:49:14.097972 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:14.097795 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl9dc\" (UniqueName: \"kubernetes.io/projected/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-kube-api-access-wl9dc\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:49:14.097972 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:14.097811 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-token\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:49:14.097972 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:14.097822 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1370edf-d7bf-42c1-b14a-1b0f4617fe27-istio-envoy\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:49:14.165674 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:14.165639 2576 generic.go:356] "Generic (PLEG): container finished" podID="e1370edf-d7bf-42c1-b14a-1b0f4617fe27" containerID="056b98ec2f43d27f807c1a0a8e7320c1096458a428980551f4757d451404f015" exitCode=0 Jul 14 19:49:14.165816 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:14.165694 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" Jul 14 19:49:14.165816 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:14.165716 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" event={"ID":"e1370edf-d7bf-42c1-b14a-1b0f4617fe27","Type":"ContainerDied","Data":"056b98ec2f43d27f807c1a0a8e7320c1096458a428980551f4757d451404f015"} Jul 14 19:49:14.165816 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:14.165769 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf" event={"ID":"e1370edf-d7bf-42c1-b14a-1b0f4617fe27","Type":"ContainerDied","Data":"151401012a2e16e17478efa0d10ee40b7f203be00f734c1d7dd69227115686d7"} Jul 14 19:49:14.165816 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:14.165790 2576 scope.go:117] "RemoveContainer" containerID="056b98ec2f43d27f807c1a0a8e7320c1096458a428980551f4757d451404f015" Jul 14 19:49:14.175645 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:14.175622 2576 scope.go:117] "RemoveContainer" containerID="9713f6b7fe181db309061615e3c6f0378438cc3c47e81607d61ae977cd7ed04a" Jul 14 19:49:14.186881 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:14.186858 2576 scope.go:117] "RemoveContainer" containerID="056b98ec2f43d27f807c1a0a8e7320c1096458a428980551f4757d451404f015" Jul 14 19:49:14.187172 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:49:14.187148 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"056b98ec2f43d27f807c1a0a8e7320c1096458a428980551f4757d451404f015\": container with ID starting with 056b98ec2f43d27f807c1a0a8e7320c1096458a428980551f4757d451404f015 not found: ID does not exist" containerID="056b98ec2f43d27f807c1a0a8e7320c1096458a428980551f4757d451404f015" Jul 14 19:49:14.187244 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:14.187180 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"056b98ec2f43d27f807c1a0a8e7320c1096458a428980551f4757d451404f015"} err="failed to get container status \"056b98ec2f43d27f807c1a0a8e7320c1096458a428980551f4757d451404f015\": rpc error: code = NotFound desc = could not find container \"056b98ec2f43d27f807c1a0a8e7320c1096458a428980551f4757d451404f015\": container with ID starting with 056b98ec2f43d27f807c1a0a8e7320c1096458a428980551f4757d451404f015 not found: ID does not exist" Jul 14 19:49:14.187244 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:14.187203 2576 scope.go:117] "RemoveContainer" containerID="9713f6b7fe181db309061615e3c6f0378438cc3c47e81607d61ae977cd7ed04a" Jul 14 19:49:14.187479 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:49:14.187458 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9713f6b7fe181db309061615e3c6f0378438cc3c47e81607d61ae977cd7ed04a\": container with ID starting with 9713f6b7fe181db309061615e3c6f0378438cc3c47e81607d61ae977cd7ed04a not found: ID does not exist" containerID="9713f6b7fe181db309061615e3c6f0378438cc3c47e81607d61ae977cd7ed04a" Jul 14 19:49:14.187540 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:14.187486 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9713f6b7fe181db309061615e3c6f0378438cc3c47e81607d61ae977cd7ed04a"} err="failed to get container status \"9713f6b7fe181db309061615e3c6f0378438cc3c47e81607d61ae977cd7ed04a\": rpc error: code = NotFound desc = could not find container \"9713f6b7fe181db309061615e3c6f0378438cc3c47e81607d61ae977cd7ed04a\": container with ID starting with 9713f6b7fe181db309061615e3c6f0378438cc3c47e81607d61ae977cd7ed04a not found: ID does not exist" Jul 14 19:49:14.203272 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:14.203246 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf"] Jul 14 19:49:14.215755 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:14.215736 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ea441188-openshift-default-d997bf747-f8prf"] Jul 14 19:49:15.514747 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:15.514715 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1370edf-d7bf-42c1-b14a-1b0f4617fe27" path="/var/lib/kubelet/pods/e1370edf-d7bf-42c1-b14a-1b0f4617fe27/volumes" Jul 14 19:49:47.670774 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:49:47.670698 2576 scope.go:117] "RemoveContainer" containerID="ce68af6f0113dbc61fece03781f26ec06d3cf4a4de0a73a97103ed29ca52d359" Jul 14 19:50:05.493295 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.493258 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5"] Jul 14 19:50:05.493743 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.493662 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1370edf-d7bf-42c1-b14a-1b0f4617fe27" containerName="istio-proxy" Jul 14 19:50:05.493743 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.493675 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1370edf-d7bf-42c1-b14a-1b0f4617fe27" containerName="istio-proxy" Jul 14 19:50:05.493743 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.493688 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f83598f-2c57-46b1-abe7-87aef32d89a6" containerName="revoke-keys" Jul 14 19:50:05.493743 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.493695 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f83598f-2c57-46b1-abe7-87aef32d89a6" containerName="revoke-keys" Jul 14 19:50:05.493743 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.493706 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1370edf-d7bf-42c1-b14a-1b0f4617fe27" containerName="istio-proxy" Jul 14 19:50:05.493743 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.493712 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1370edf-d7bf-42c1-b14a-1b0f4617fe27" containerName="istio-proxy" Jul 14 19:50:05.494358 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.493782 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f83598f-2c57-46b1-abe7-87aef32d89a6" containerName="revoke-keys" Jul 14 19:50:05.494358 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.493804 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1370edf-d7bf-42c1-b14a-1b0f4617fe27" containerName="istio-proxy" Jul 14 19:50:05.497071 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.497048 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.500490 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.500468 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0c147a-openshift-default-dockercfg-569sm\"" Jul 14 19:50:05.518222 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.518192 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5"] Jul 14 19:50:05.587981 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.587950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-token\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.588159 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.588005 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-workload-socket\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.588159 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.588056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-workload-certs\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.588159 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.588083 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-envoy\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.588159 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.588114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gh4x\" (UniqueName: \"kubernetes.io/projected/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-kube-api-access-6gh4x\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.588159 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.588143 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-data\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.588356 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.588170 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istiod-ca-cert\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.588356 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.588220 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-credential-socket\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.588432 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.588396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-podinfo\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.689057 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.689015 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-data\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.689237 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.689069 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istiod-ca-cert\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.689237 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.689123 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-credential-socket\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.689237 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.689176 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-podinfo\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.689237 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.689201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-token\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.689454 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.689361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-workload-socket\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.689454 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.689432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-workload-certs\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.689454 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.689439 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-data\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.689627 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.689469 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-envoy\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.689627 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.689511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gh4x\" (UniqueName: \"kubernetes.io/projected/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-kube-api-access-6gh4x\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.689757 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.689695 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-credential-socket\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.689757 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.689735 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-workload-socket\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.689860 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.689801 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istiod-ca-cert\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.689991 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.689926 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-workload-certs\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.692435 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.692410 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-podinfo\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.692673 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.692654 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-envoy\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.700231 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.700203 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-token\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.700335 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.700249 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gh4x\" (UniqueName: \"kubernetes.io/projected/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-kube-api-access-6gh4x\") pod \"e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.808786 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.808703 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:05.960403 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.960372 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5"] Jul 14 19:50:05.963377 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:50:05.963346 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a502a48_5ad0_4db8_a7a5_6990bbaabf52.slice/crio-ea79b46661de30e814f8348b90b743db7182ad6b3c68748ba006c48d87c573e5 WatchSource:0}: Error finding container ea79b46661de30e814f8348b90b743db7182ad6b3c68748ba006c48d87c573e5: Status 404 returned error can't find the container with id ea79b46661de30e814f8348b90b743db7182ad6b3c68748ba006c48d87c573e5 Jul 14 19:50:05.965842 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.965806 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:50:05.965939 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.965875 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:50:05.965939 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:05.965909 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:50:06.431175 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:06.431142 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" event={"ID":"0a502a48-5ad0-4db8-a7a5-6990bbaabf52","Type":"ContainerStarted","Data":"ea3107f3b5fbd1a2708570e8327fca2b2371ea402cceac4876bdf353ac11b9d4"} Jul 14 19:50:06.431352 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:06.431195 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" event={"ID":"0a502a48-5ad0-4db8-a7a5-6990bbaabf52","Type":"ContainerStarted","Data":"ea79b46661de30e814f8348b90b743db7182ad6b3c68748ba006c48d87c573e5"} Jul 14 19:50:06.456796 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:06.456744 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" podStartSLOduration=1.456726271 podStartE2EDuration="1.456726271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:50:06.454631338 +0000 UTC m=+2359.567881211" watchObservedRunningTime="2026-07-14 19:50:06.456726271 +0000 UTC m=+2359.569976140" Jul 14 19:50:06.808851 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:06.808764 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:07.814408 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:07.814379 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:08.442557 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:08.442521 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:08.443601 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:08.443568 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:13.663051 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.663013 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf"] Jul 14 19:50:13.663560 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.663546 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1370edf-d7bf-42c1-b14a-1b0f4617fe27" containerName="istio-proxy" Jul 14 19:50:13.667022 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.667006 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.669856 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.669836 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7fc5dd-openshift-default-dockercfg-fggtm\"" Jul 14 19:50:13.679931 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.679906 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf"] Jul 14 19:50:13.778290 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.778251 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-istio-envoy\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.778290 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.778300 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433f779c-ff71-466b-87eb-895b4bf6458e-istio-token\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.778527 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.778345 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-workload-certs\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.778527 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.778384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-workload-socket\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.778527 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.778471 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433f779c-ff71-466b-87eb-895b4bf6458e-istiod-ca-cert\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.778527 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.778501 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d9xm\" (UniqueName: \"kubernetes.io/projected/433f779c-ff71-466b-87eb-895b4bf6458e-kube-api-access-8d9xm\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.778693 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.778531 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-credential-socket\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.778693 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.778575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-istio-data\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.778693 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.778633 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433f779c-ff71-466b-87eb-895b4bf6458e-istio-podinfo\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.879388 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.879348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-workload-certs\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.879602 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.879400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-workload-socket\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.879602 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.879458 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433f779c-ff71-466b-87eb-895b4bf6458e-istiod-ca-cert\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.879602 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.879486 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d9xm\" (UniqueName: \"kubernetes.io/projected/433f779c-ff71-466b-87eb-895b4bf6458e-kube-api-access-8d9xm\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.879602 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.879512 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-credential-socket\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.879602 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.879558 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-istio-data\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.879886 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.879618 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433f779c-ff71-466b-87eb-895b4bf6458e-istio-podinfo\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.879886 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.879669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-istio-envoy\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.879886 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.879697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433f779c-ff71-466b-87eb-895b4bf6458e-istio-token\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.879886 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.879869 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-workload-socket\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.880130 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.880101 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-credential-socket\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.880270 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.880247 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433f779c-ff71-466b-87eb-895b4bf6458e-istiod-ca-cert\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.880361 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.880341 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-istio-data\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.880419 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.880354 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-workload-certs\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.882118 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.882087 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-istio-envoy\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.882313 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.882291 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433f779c-ff71-466b-87eb-895b4bf6458e-istio-podinfo\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.889149 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.889123 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433f779c-ff71-466b-87eb-895b4bf6458e-istio-token\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.889427 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.889406 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d9xm\" (UniqueName: \"kubernetes.io/projected/433f779c-ff71-466b-87eb-895b4bf6458e-kube-api-access-8d9xm\") pod \"e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:13.980133 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:13.980027 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:14.125955 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:14.125930 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf"] Jul 14 19:50:14.127126 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:50:14.127097 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433f779c_ff71_466b_87eb_895b4bf6458e.slice/crio-34b500fca0e2a5e901acbde3097622853f09116fbf48dafa568d76e55c4d2f48 WatchSource:0}: Error finding container 34b500fca0e2a5e901acbde3097622853f09116fbf48dafa568d76e55c4d2f48: Status 404 returned error can't find the container with id 34b500fca0e2a5e901acbde3097622853f09116fbf48dafa568d76e55c4d2f48 Jul 14 19:50:14.129626 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:14.129571 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:50:14.129707 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:14.129659 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:50:14.129707 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:14.129685 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:50:14.474962 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:14.474921 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" event={"ID":"433f779c-ff71-466b-87eb-895b4bf6458e","Type":"ContainerStarted","Data":"8808b9aa3bd05453291d680acc558fcc72af370e49e4c7446704831027dc8709"} Jul 14 19:50:14.475119 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:14.474976 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" event={"ID":"433f779c-ff71-466b-87eb-895b4bf6458e","Type":"ContainerStarted","Data":"34b500fca0e2a5e901acbde3097622853f09116fbf48dafa568d76e55c4d2f48"} Jul 14 19:50:14.498457 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:14.498403 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" podStartSLOduration=1.498386657 podStartE2EDuration="1.498386657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:50:14.497396736 +0000 UTC m=+2367.610646606" watchObservedRunningTime="2026-07-14 19:50:14.498386657 +0000 UTC m=+2367.611636628" Jul 14 19:50:14.980902 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:14.980869 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:15.986220 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:15.986192 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:16.486992 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:16.486953 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:16.488114 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:16.488090 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:24.464084 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:24.464027 2576 patch_prober.go:28] interesting pod/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 19:50:24.464538 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:24.464245 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" podUID="0a502a48-5ad0-4db8-a7a5-6990bbaabf52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 19:50:24.541072 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:24.541038 2576 generic.go:356] "Generic (PLEG): container finished" podID="0a502a48-5ad0-4db8-a7a5-6990bbaabf52" containerID="ea3107f3b5fbd1a2708570e8327fca2b2371ea402cceac4876bdf353ac11b9d4" exitCode=0 Jul 14 19:50:24.541238 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:24.541100 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" event={"ID":"0a502a48-5ad0-4db8-a7a5-6990bbaabf52","Type":"ContainerDied","Data":"ea3107f3b5fbd1a2708570e8327fca2b2371ea402cceac4876bdf353ac11b9d4"} Jul 14 19:50:24.541521 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:24.541506 2576 scope.go:117] "RemoveContainer" containerID="ea3107f3b5fbd1a2708570e8327fca2b2371ea402cceac4876bdf353ac11b9d4" Jul 14 19:50:24.542630 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:24.542593 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:50:24.542705 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:24.542689 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:50:24.542747 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:24.542730 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:50:25.550631 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:25.550593 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" event={"ID":"0a502a48-5ad0-4db8-a7a5-6990bbaabf52","Type":"ContainerStarted","Data":"acdad9286666a0d62c184ad9d55fab8c4afe252f26431a1a6338e786dfeef393"} Jul 14 19:50:25.809402 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:25.809283 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:25.809402 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:25.809368 2576 patch_prober.go:28] interesting pod/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 14 19:50:25.809686 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:25.809438 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" podUID="0a502a48-5ad0-4db8-a7a5-6990bbaabf52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 14 19:50:26.809971 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:26.809923 2576 patch_prober.go:28] interesting pod/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 14 19:50:26.810480 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:26.809995 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" podUID="0a502a48-5ad0-4db8-a7a5-6990bbaabf52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 14 19:50:27.809724 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:27.809678 2576 patch_prober.go:28] interesting pod/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 14 19:50:27.809949 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:27.809765 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" podUID="0a502a48-5ad0-4db8-a7a5-6990bbaabf52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 14 19:50:28.809406 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:28.809357 2576 patch_prober.go:28] interesting pod/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 14 19:50:28.809960 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:28.809438 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" podUID="0a502a48-5ad0-4db8-a7a5-6990bbaabf52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 14 19:50:29.814103 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:29.814067 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:29.814569 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:29.814486 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:29.815286 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:29.815264 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:50:32.518422 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:32.518375 2576 patch_prober.go:28] interesting pod/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 19:50:32.518857 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:32.518460 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" podUID="433f779c-ff71-466b-87eb-895b4bf6458e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 19:50:32.594317 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:32.594285 2576 generic.go:356] "Generic (PLEG): container finished" podID="433f779c-ff71-466b-87eb-895b4bf6458e" containerID="8808b9aa3bd05453291d680acc558fcc72af370e49e4c7446704831027dc8709" exitCode=0 Jul 14 19:50:32.594477 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:32.594339 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" event={"ID":"433f779c-ff71-466b-87eb-895b4bf6458e","Type":"ContainerDied","Data":"8808b9aa3bd05453291d680acc558fcc72af370e49e4c7446704831027dc8709"} Jul 14 19:50:32.594759 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:32.594742 2576 scope.go:117] "RemoveContainer" containerID="8808b9aa3bd05453291d680acc558fcc72af370e49e4c7446704831027dc8709" Jul 14 19:50:32.595803 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:32.595772 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:50:32.595896 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:32.595858 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:50:32.595965 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:32.595910 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:50:33.605080 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:33.605045 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" event={"ID":"433f779c-ff71-466b-87eb-895b4bf6458e","Type":"ContainerStarted","Data":"a13820553454f31a977aa6feedfe0ebe028fcf25849e639556582749dbc3e133"} Jul 14 19:50:33.981034 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:33.980927 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:33.981231 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:33.981070 2576 patch_prober.go:28] interesting pod/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 14 19:50:33.981231 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:33.981127 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" podUID="433f779c-ff71-466b-87eb-895b4bf6458e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 14 19:50:34.981019 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:34.980970 2576 patch_prober.go:28] interesting pod/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 14 19:50:34.981617 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:34.981559 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" podUID="433f779c-ff71-466b-87eb-895b4bf6458e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 14 19:50:35.981011 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:35.980962 2576 patch_prober.go:28] interesting pod/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 14 19:50:35.981334 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:35.981280 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" podUID="433f779c-ff71-466b-87eb-895b4bf6458e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 14 19:50:36.981361 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:36.981313 2576 patch_prober.go:28] interesting pod/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 14 19:50:36.981906 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:36.981387 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" podUID="433f779c-ff71-466b-87eb-895b4bf6458e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 14 19:50:37.984780 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:37.984750 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:37.985216 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:37.985086 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:37.985889 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:37.985868 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:50:47.594985 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:47.594954 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:50:47.605294 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:50:47.605270 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:51:41.440985 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:41.440950 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5"] Jul 14 19:51:41.441486 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:41.441224 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" podUID="0a502a48-5ad0-4db8-a7a5-6990bbaabf52" containerName="istio-proxy" containerID="cri-o://acdad9286666a0d62c184ad9d55fab8c4afe252f26431a1a6338e786dfeef393" gracePeriod=30 Jul 14 19:51:44.816934 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:44.816898 2576 patch_prober.go:28] interesting pod/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 19:51:44.817324 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:44.816955 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" podUID="0a502a48-5ad0-4db8-a7a5-6990bbaabf52" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 19:51:46.696161 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.696133 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:51:46.723624 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.723505 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-envoy\") pod \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " Jul 14 19:51:46.723624 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.723569 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-credential-socket\") pod \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " Jul 14 19:51:46.723624 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.723607 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-data\") pod \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " Jul 14 19:51:46.723911 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.723642 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-workload-socket\") pod \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " Jul 14 19:51:46.723911 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.723666 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gh4x\" (UniqueName: \"kubernetes.io/projected/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-kube-api-access-6gh4x\") pod \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " Jul 14 19:51:46.723911 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.723711 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istiod-ca-cert\") pod \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " Jul 14 19:51:46.723911 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.723738 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-podinfo\") pod \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " Jul 14 19:51:46.723911 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.723768 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-workload-certs\") pod \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " Jul 14 19:51:46.723911 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.723814 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-token\") pod \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\" (UID: \"0a502a48-5ad0-4db8-a7a5-6990bbaabf52\") " Jul 14 19:51:46.724216 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.723949 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0a502a48-5ad0-4db8-a7a5-6990bbaabf52" (UID: "0a502a48-5ad0-4db8-a7a5-6990bbaabf52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:46.724216 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.724168 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-credential-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:51:46.724329 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.724210 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0a502a48-5ad0-4db8-a7a5-6990bbaabf52" (UID: "0a502a48-5ad0-4db8-a7a5-6990bbaabf52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:51:46.724329 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.724251 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0a502a48-5ad0-4db8-a7a5-6990bbaabf52" (UID: "0a502a48-5ad0-4db8-a7a5-6990bbaabf52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:46.724521 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.724497 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0a502a48-5ad0-4db8-a7a5-6990bbaabf52" (UID: "0a502a48-5ad0-4db8-a7a5-6990bbaabf52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:46.724692 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.724661 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-data" (OuterVolumeSpecName: "istio-data") pod "0a502a48-5ad0-4db8-a7a5-6990bbaabf52" (UID: "0a502a48-5ad0-4db8-a7a5-6990bbaabf52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:46.726304 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.726274 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0a502a48-5ad0-4db8-a7a5-6990bbaabf52" (UID: "0a502a48-5ad0-4db8-a7a5-6990bbaabf52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:46.726538 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.726509 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0a502a48-5ad0-4db8-a7a5-6990bbaabf52" (UID: "0a502a48-5ad0-4db8-a7a5-6990bbaabf52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 19:51:46.726778 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.726758 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-token" (OuterVolumeSpecName: "istio-token") pod "0a502a48-5ad0-4db8-a7a5-6990bbaabf52" (UID: "0a502a48-5ad0-4db8-a7a5-6990bbaabf52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:51:46.726868 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.726787 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-kube-api-access-6gh4x" (OuterVolumeSpecName: "kube-api-access-6gh4x") pod "0a502a48-5ad0-4db8-a7a5-6990bbaabf52" (UID: "0a502a48-5ad0-4db8-a7a5-6990bbaabf52"). InnerVolumeSpecName "kube-api-access-6gh4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:51:46.824707 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.824669 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istiod-ca-cert\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:51:46.824707 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.824700 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-podinfo\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:51:46.824707 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.824710 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-workload-certs\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:51:46.824707 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.824719 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-token\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:51:46.824969 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.824727 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-envoy\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:51:46.824969 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.824734 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-istio-data\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:51:46.824969 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.824741 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-workload-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:51:46.824969 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:46.824749 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gh4x\" (UniqueName: \"kubernetes.io/projected/0a502a48-5ad0-4db8-a7a5-6990bbaabf52-kube-api-access-6gh4x\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:51:47.013206 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:47.013126 2576 generic.go:356] "Generic (PLEG): container finished" podID="0a502a48-5ad0-4db8-a7a5-6990bbaabf52" containerID="acdad9286666a0d62c184ad9d55fab8c4afe252f26431a1a6338e786dfeef393" exitCode=0 Jul 14 19:51:47.013206 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:47.013176 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" Jul 14 19:51:47.013206 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:47.013183 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" event={"ID":"0a502a48-5ad0-4db8-a7a5-6990bbaabf52","Type":"ContainerDied","Data":"acdad9286666a0d62c184ad9d55fab8c4afe252f26431a1a6338e786dfeef393"} Jul 14 19:51:47.013436 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:47.013227 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5" event={"ID":"0a502a48-5ad0-4db8-a7a5-6990bbaabf52","Type":"ContainerDied","Data":"ea79b46661de30e814f8348b90b743db7182ad6b3c68748ba006c48d87c573e5"} Jul 14 19:51:47.013436 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:47.013247 2576 scope.go:117] "RemoveContainer" containerID="acdad9286666a0d62c184ad9d55fab8c4afe252f26431a1a6338e786dfeef393" Jul 14 19:51:47.022559 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:47.022540 2576 scope.go:117] "RemoveContainer" containerID="ea3107f3b5fbd1a2708570e8327fca2b2371ea402cceac4876bdf353ac11b9d4" Jul 14 19:51:47.031122 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:47.031101 2576 scope.go:117] "RemoveContainer" containerID="acdad9286666a0d62c184ad9d55fab8c4afe252f26431a1a6338e786dfeef393" Jul 14 19:51:47.031360 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:51:47.031338 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acdad9286666a0d62c184ad9d55fab8c4afe252f26431a1a6338e786dfeef393\": container with ID starting with acdad9286666a0d62c184ad9d55fab8c4afe252f26431a1a6338e786dfeef393 not found: ID does not exist" containerID="acdad9286666a0d62c184ad9d55fab8c4afe252f26431a1a6338e786dfeef393" Jul 14 19:51:47.031406 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:47.031368 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acdad9286666a0d62c184ad9d55fab8c4afe252f26431a1a6338e786dfeef393"} err="failed to get container status \"acdad9286666a0d62c184ad9d55fab8c4afe252f26431a1a6338e786dfeef393\": rpc error: code = NotFound desc = could not find container \"acdad9286666a0d62c184ad9d55fab8c4afe252f26431a1a6338e786dfeef393\": container with ID starting with acdad9286666a0d62c184ad9d55fab8c4afe252f26431a1a6338e786dfeef393 not found: ID does not exist" Jul 14 19:51:47.031406 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:47.031385 2576 scope.go:117] "RemoveContainer" containerID="ea3107f3b5fbd1a2708570e8327fca2b2371ea402cceac4876bdf353ac11b9d4" Jul 14 19:51:47.031599 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:51:47.031567 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea3107f3b5fbd1a2708570e8327fca2b2371ea402cceac4876bdf353ac11b9d4\": container with ID starting with ea3107f3b5fbd1a2708570e8327fca2b2371ea402cceac4876bdf353ac11b9d4 not found: ID does not exist" containerID="ea3107f3b5fbd1a2708570e8327fca2b2371ea402cceac4876bdf353ac11b9d4" Jul 14 19:51:47.031668 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:47.031597 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea3107f3b5fbd1a2708570e8327fca2b2371ea402cceac4876bdf353ac11b9d4"} err="failed to get container status \"ea3107f3b5fbd1a2708570e8327fca2b2371ea402cceac4876bdf353ac11b9d4\": rpc error: code = NotFound desc = could not find container \"ea3107f3b5fbd1a2708570e8327fca2b2371ea402cceac4876bdf353ac11b9d4\": container with ID starting with ea3107f3b5fbd1a2708570e8327fca2b2371ea402cceac4876bdf353ac11b9d4 not found: ID does not exist" Jul 14 19:51:47.039113 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:47.039088 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5"] Jul 14 19:51:47.042820 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:47.042795 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0c147a-openshift-default-64f89f48f-bn7g5"] Jul 14 19:51:47.515973 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:51:47.515944 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a502a48-5ad0-4db8-a7a5-6990bbaabf52" path="/var/lib/kubelet/pods/0a502a48-5ad0-4db8-a7a5-6990bbaabf52/volumes" Jul 14 19:52:07.612197 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:07.612160 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf"] Jul 14 19:52:07.612718 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:07.612388 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" podUID="433f779c-ff71-466b-87eb-895b4bf6458e" containerName="istio-proxy" containerID="cri-o://a13820553454f31a977aa6feedfe0ebe028fcf25849e639556582749dbc3e133" gracePeriod=30 Jul 14 19:52:07.987991 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:07.987908 2576 patch_prober.go:28] interesting pod/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 19:52:07.988135 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:07.987980 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" podUID="433f779c-ff71-466b-87eb-895b4bf6458e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 19:52:08.560208 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.560175 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9"] Jul 14 19:52:08.560618 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.560572 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a502a48-5ad0-4db8-a7a5-6990bbaabf52" containerName="istio-proxy" Jul 14 19:52:08.560618 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.560619 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a502a48-5ad0-4db8-a7a5-6990bbaabf52" containerName="istio-proxy" Jul 14 19:52:08.560767 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.560694 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a502a48-5ad0-4db8-a7a5-6990bbaabf52" containerName="istio-proxy" Jul 14 19:52:08.560767 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.560702 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a502a48-5ad0-4db8-a7a5-6990bbaabf52" containerName="istio-proxy" Jul 14 19:52:08.563055 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.563033 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.565776 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.565759 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-922b98-openshift-default-dockercfg-c46sg\"" Jul 14 19:52:08.591521 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.591487 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9"] Jul 14 19:52:08.623538 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.623501 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-credential-socket\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.624029 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.623563 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ccf82-87f6-4332-8838-46f449d15c52-istiod-ca-cert\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.624029 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.623724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-data\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.624029 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.623771 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-podinfo\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.624029 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.623798 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt2wt\" (UniqueName: \"kubernetes.io/projected/1d6ccf82-87f6-4332-8838-46f449d15c52-kube-api-access-jt2wt\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.624029 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.623916 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-token\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.624029 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.623966 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-workload-socket\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.624330 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.624036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-workload-certs\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.624330 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.624168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-envoy\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.725560 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.725522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-data\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.725560 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.725566 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-podinfo\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.725831 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.725614 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt2wt\" (UniqueName: \"kubernetes.io/projected/1d6ccf82-87f6-4332-8838-46f449d15c52-kube-api-access-jt2wt\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.725831 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.725663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-token\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.725831 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.725688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-workload-socket\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.725831 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.725758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-workload-certs\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.725831 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.725789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-envoy\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.725831 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.725832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-credential-socket\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.726142 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.725867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ccf82-87f6-4332-8838-46f449d15c52-istiod-ca-cert\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.726201 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.726145 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-workload-socket\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.726201 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.726176 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-workload-certs\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.726357 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.726310 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-credential-socket\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.726458 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.726419 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-data\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.726576 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.726556 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ccf82-87f6-4332-8838-46f449d15c52-istiod-ca-cert\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.728528 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.728503 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-envoy\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.728873 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.728852 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-podinfo\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.736471 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.736421 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-token\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.736595 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.736519 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt2wt\" (UniqueName: \"kubernetes.io/projected/1d6ccf82-87f6-4332-8838-46f449d15c52-kube-api-access-jt2wt\") pod \"e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:08.876862 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:08.876824 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:09.031098 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:09.031063 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9"] Jul 14 19:52:09.033016 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:52:09.032990 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6ccf82_87f6_4332_8838_46f449d15c52.slice/crio-6c517d9c751181c58fbfae007d426aca3b4d7326bce9ccbed14bf9174141acff WatchSource:0}: Error finding container 6c517d9c751181c58fbfae007d426aca3b4d7326bce9ccbed14bf9174141acff: Status 404 returned error can't find the container with id 6c517d9c751181c58fbfae007d426aca3b4d7326bce9ccbed14bf9174141acff Jul 14 19:52:09.035681 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:09.035638 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:52:09.035764 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:09.035731 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:52:09.035814 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:09.035774 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:52:09.132074 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:09.131916 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" event={"ID":"1d6ccf82-87f6-4332-8838-46f449d15c52","Type":"ContainerStarted","Data":"25f1367a38602fc560f51f4362d67defc6f4e78d7b929a087d678e9ef25d7c7a"} Jul 14 19:52:09.132074 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:09.131987 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" event={"ID":"1d6ccf82-87f6-4332-8838-46f449d15c52","Type":"ContainerStarted","Data":"6c517d9c751181c58fbfae007d426aca3b4d7326bce9ccbed14bf9174141acff"} Jul 14 19:52:09.160957 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:09.159670 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" podStartSLOduration=1.159645817 podStartE2EDuration="1.159645817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:09.158475019 +0000 UTC m=+2482.271724890" watchObservedRunningTime="2026-07-14 19:52:09.159645817 +0000 UTC m=+2482.272895691" Jul 14 19:52:09.877576 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:09.877541 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:10.882314 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:10.882285 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:11.146210 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:11.146125 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:11.147221 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:11.147203 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:12.882353 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.882291 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:52:12.970941 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.970905 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433f779c-ff71-466b-87eb-895b4bf6458e-istiod-ca-cert\") pod \"433f779c-ff71-466b-87eb-895b4bf6458e\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " Jul 14 19:52:12.971124 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.970966 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433f779c-ff71-466b-87eb-895b4bf6458e-istio-podinfo\") pod \"433f779c-ff71-466b-87eb-895b4bf6458e\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " Jul 14 19:52:12.971124 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.971003 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-workload-socket\") pod \"433f779c-ff71-466b-87eb-895b4bf6458e\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " Jul 14 19:52:12.971124 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.971031 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433f779c-ff71-466b-87eb-895b4bf6458e-istio-token\") pod \"433f779c-ff71-466b-87eb-895b4bf6458e\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " Jul 14 19:52:12.971124 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.971094 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-istio-data\") pod \"433f779c-ff71-466b-87eb-895b4bf6458e\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " Jul 14 19:52:12.971399 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.971160 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d9xm\" (UniqueName: \"kubernetes.io/projected/433f779c-ff71-466b-87eb-895b4bf6458e-kube-api-access-8d9xm\") pod \"433f779c-ff71-466b-87eb-895b4bf6458e\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " Jul 14 19:52:12.971399 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.971187 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-workload-certs\") pod \"433f779c-ff71-466b-87eb-895b4bf6458e\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " Jul 14 19:52:12.971399 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.971245 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-istio-envoy\") pod \"433f779c-ff71-466b-87eb-895b4bf6458e\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " Jul 14 19:52:12.971399 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.971295 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-credential-socket\") pod \"433f779c-ff71-466b-87eb-895b4bf6458e\" (UID: \"433f779c-ff71-466b-87eb-895b4bf6458e\") " Jul 14 19:52:12.971399 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.971287 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "433f779c-ff71-466b-87eb-895b4bf6458e" (UID: "433f779c-ff71-466b-87eb-895b4bf6458e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:52:12.971399 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.971346 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433f779c-ff71-466b-87eb-895b4bf6458e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "433f779c-ff71-466b-87eb-895b4bf6458e" (UID: "433f779c-ff71-466b-87eb-895b4bf6458e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:52:12.971747 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.971613 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433f779c-ff71-466b-87eb-895b4bf6458e-istiod-ca-cert\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:52:12.971747 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.971631 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-workload-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:52:12.971859 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.971755 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "433f779c-ff71-466b-87eb-895b4bf6458e" (UID: "433f779c-ff71-466b-87eb-895b4bf6458e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:52:12.971859 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.971769 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "433f779c-ff71-466b-87eb-895b4bf6458e" (UID: "433f779c-ff71-466b-87eb-895b4bf6458e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:52:12.972482 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.972459 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-istio-data" (OuterVolumeSpecName: "istio-data") pod "433f779c-ff71-466b-87eb-895b4bf6458e" (UID: "433f779c-ff71-466b-87eb-895b4bf6458e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:52:12.973705 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.973607 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433f779c-ff71-466b-87eb-895b4bf6458e-istio-token" (OuterVolumeSpecName: "istio-token") pod "433f779c-ff71-466b-87eb-895b4bf6458e" (UID: "433f779c-ff71-466b-87eb-895b4bf6458e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:52:12.973926 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.973890 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/433f779c-ff71-466b-87eb-895b4bf6458e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "433f779c-ff71-466b-87eb-895b4bf6458e" (UID: "433f779c-ff71-466b-87eb-895b4bf6458e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 19:52:12.974006 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.973968 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433f779c-ff71-466b-87eb-895b4bf6458e-kube-api-access-8d9xm" (OuterVolumeSpecName: "kube-api-access-8d9xm") pod "433f779c-ff71-466b-87eb-895b4bf6458e" (UID: "433f779c-ff71-466b-87eb-895b4bf6458e"). InnerVolumeSpecName "kube-api-access-8d9xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:52:12.974833 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:12.974800 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "433f779c-ff71-466b-87eb-895b4bf6458e" (UID: "433f779c-ff71-466b-87eb-895b4bf6458e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:52:13.072172 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.072138 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-istio-data\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:52:13.072172 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.072168 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d9xm\" (UniqueName: \"kubernetes.io/projected/433f779c-ff71-466b-87eb-895b4bf6458e-kube-api-access-8d9xm\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:52:13.072172 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.072177 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-workload-certs\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:52:13.072419 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.072186 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-istio-envoy\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:52:13.072419 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.072194 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f779c-ff71-466b-87eb-895b4bf6458e-credential-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:52:13.072419 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.072203 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433f779c-ff71-466b-87eb-895b4bf6458e-istio-podinfo\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:52:13.072419 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.072211 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433f779c-ff71-466b-87eb-895b4bf6458e-istio-token\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:52:13.158821 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.158737 2576 generic.go:356] "Generic (PLEG): container finished" podID="433f779c-ff71-466b-87eb-895b4bf6458e" containerID="a13820553454f31a977aa6feedfe0ebe028fcf25849e639556582749dbc3e133" exitCode=0 Jul 14 19:52:13.158821 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.158794 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" Jul 14 19:52:13.159002 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.158823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" event={"ID":"433f779c-ff71-466b-87eb-895b4bf6458e","Type":"ContainerDied","Data":"a13820553454f31a977aa6feedfe0ebe028fcf25849e639556582749dbc3e133"} Jul 14 19:52:13.159002 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.158882 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf" event={"ID":"433f779c-ff71-466b-87eb-895b4bf6458e","Type":"ContainerDied","Data":"34b500fca0e2a5e901acbde3097622853f09116fbf48dafa568d76e55c4d2f48"} Jul 14 19:52:13.159002 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.158912 2576 scope.go:117] "RemoveContainer" containerID="a13820553454f31a977aa6feedfe0ebe028fcf25849e639556582749dbc3e133" Jul 14 19:52:13.168594 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.168561 2576 scope.go:117] "RemoveContainer" containerID="8808b9aa3bd05453291d680acc558fcc72af370e49e4c7446704831027dc8709" Jul 14 19:52:13.177150 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.177129 2576 scope.go:117] "RemoveContainer" containerID="a13820553454f31a977aa6feedfe0ebe028fcf25849e639556582749dbc3e133" Jul 14 19:52:13.177449 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:52:13.177432 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a13820553454f31a977aa6feedfe0ebe028fcf25849e639556582749dbc3e133\": container with ID starting with a13820553454f31a977aa6feedfe0ebe028fcf25849e639556582749dbc3e133 not found: ID does not exist" containerID="a13820553454f31a977aa6feedfe0ebe028fcf25849e639556582749dbc3e133" Jul 14 19:52:13.177500 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.177459 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13820553454f31a977aa6feedfe0ebe028fcf25849e639556582749dbc3e133"} err="failed to get container status \"a13820553454f31a977aa6feedfe0ebe028fcf25849e639556582749dbc3e133\": rpc error: code = NotFound desc = could not find container \"a13820553454f31a977aa6feedfe0ebe028fcf25849e639556582749dbc3e133\": container with ID starting with a13820553454f31a977aa6feedfe0ebe028fcf25849e639556582749dbc3e133 not found: ID does not exist" Jul 14 19:52:13.177500 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.177477 2576 scope.go:117] "RemoveContainer" containerID="8808b9aa3bd05453291d680acc558fcc72af370e49e4c7446704831027dc8709" Jul 14 19:52:13.177750 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:52:13.177732 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8808b9aa3bd05453291d680acc558fcc72af370e49e4c7446704831027dc8709\": container with ID starting with 8808b9aa3bd05453291d680acc558fcc72af370e49e4c7446704831027dc8709 not found: ID does not exist" containerID="8808b9aa3bd05453291d680acc558fcc72af370e49e4c7446704831027dc8709" Jul 14 19:52:13.177806 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.177755 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8808b9aa3bd05453291d680acc558fcc72af370e49e4c7446704831027dc8709"} err="failed to get container status \"8808b9aa3bd05453291d680acc558fcc72af370e49e4c7446704831027dc8709\": rpc error: code = NotFound desc = could not find container \"8808b9aa3bd05453291d680acc558fcc72af370e49e4c7446704831027dc8709\": container with ID starting with 8808b9aa3bd05453291d680acc558fcc72af370e49e4c7446704831027dc8709 not found: ID does not exist" Jul 14 19:52:13.181745 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.181721 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf"] Jul 14 19:52:13.187126 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.187100 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7fc5dd-openshift-default-7b8cd77885-gg9kf"] Jul 14 19:52:13.516467 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:13.516376 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="433f779c-ff71-466b-87eb-895b4bf6458e" path="/var/lib/kubelet/pods/433f779c-ff71-466b-87eb-895b4bf6458e/volumes" Jul 14 19:52:16.510184 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.510151 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt"] Jul 14 19:52:16.510664 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.510544 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="433f779c-ff71-466b-87eb-895b4bf6458e" containerName="istio-proxy" Jul 14 19:52:16.510664 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.510555 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="433f779c-ff71-466b-87eb-895b4bf6458e" containerName="istio-proxy" Jul 14 19:52:16.510664 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.510564 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a502a48-5ad0-4db8-a7a5-6990bbaabf52" containerName="istio-proxy" Jul 14 19:52:16.510664 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.510571 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a502a48-5ad0-4db8-a7a5-6990bbaabf52" containerName="istio-proxy" Jul 14 19:52:16.510887 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.510672 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="433f779c-ff71-466b-87eb-895b4bf6458e" containerName="istio-proxy" Jul 14 19:52:16.515319 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.515300 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.518761 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.518739 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-57e988-openshift-default-dockercfg-rcrq5\"" Jul 14 19:52:16.530665 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.530646 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt"] Jul 14 19:52:16.605159 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.605119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-workload-socket\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.605346 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.605170 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-credential-socket\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.605346 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.605310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-workload-certs\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.605458 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.605347 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98vjt\" (UniqueName: \"kubernetes.io/projected/eb68a821-ade5-40f2-8691-81a85a8e9373-kube-api-access-98vjt\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.605458 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.605385 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-token\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.605458 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.605447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb68a821-ade5-40f2-8691-81a85a8e9373-istiod-ca-cert\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.605951 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.605487 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-podinfo\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.605951 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.605555 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-data\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.605951 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.605638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-envoy\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.706931 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.706903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-workload-socket\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.706931 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.706937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-credential-socket\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.707184 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.706970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-workload-certs\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.707184 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.707007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98vjt\" (UniqueName: \"kubernetes.io/projected/eb68a821-ade5-40f2-8691-81a85a8e9373-kube-api-access-98vjt\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.707184 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.707028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-token\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.707184 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.707065 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb68a821-ade5-40f2-8691-81a85a8e9373-istiod-ca-cert\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.707184 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.707108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-podinfo\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.707453 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.707278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-data\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.707453 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.707333 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-workload-certs\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.707453 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.707345 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-envoy\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.707453 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.707393 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-workload-socket\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.707698 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.707615 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-credential-socket\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.707698 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.707680 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-data\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.707849 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.707824 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb68a821-ade5-40f2-8691-81a85a8e9373-istiod-ca-cert\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.709439 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.709409 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-envoy\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.709550 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.709423 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-podinfo\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.716116 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.716093 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-token\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.716220 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.716202 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98vjt\" (UniqueName: \"kubernetes.io/projected/eb68a821-ade5-40f2-8691-81a85a8e9373-kube-api-access-98vjt\") pod \"e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.827746 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.827712 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:16.977496 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.977471 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt"] Jul 14 19:52:16.978804 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:52:16.978778 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb68a821_ade5_40f2_8691_81a85a8e9373.slice/crio-4964c2aad7bbed29a1b88fc3d533396b73d4e9896b3afb85c6cad1c5988d3983 WatchSource:0}: Error finding container 4964c2aad7bbed29a1b88fc3d533396b73d4e9896b3afb85c6cad1c5988d3983: Status 404 returned error can't find the container with id 4964c2aad7bbed29a1b88fc3d533396b73d4e9896b3afb85c6cad1c5988d3983 Jul 14 19:52:16.981337 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.981308 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:52:16.981421 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.981371 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:52:16.981421 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:16.981399 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:52:17.184062 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:17.183960 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" event={"ID":"eb68a821-ade5-40f2-8691-81a85a8e9373","Type":"ContainerStarted","Data":"60bdada8e90fdc8a1c34f51b8d20ec0751d892502d5a27735a956d1dcca17faf"} Jul 14 19:52:17.184062 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:17.184022 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" event={"ID":"eb68a821-ade5-40f2-8691-81a85a8e9373","Type":"ContainerStarted","Data":"4964c2aad7bbed29a1b88fc3d533396b73d4e9896b3afb85c6cad1c5988d3983"} Jul 14 19:52:17.209006 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:17.208950 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" podStartSLOduration=1.20893423 podStartE2EDuration="1.20893423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:17.206461967 +0000 UTC m=+2490.319711838" watchObservedRunningTime="2026-07-14 19:52:17.20893423 +0000 UTC m=+2490.322184100" Jul 14 19:52:17.828685 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:17.828654 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:18.833388 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:18.833358 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:19.195304 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:19.195199 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:19.196337 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:19.196316 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:26.550174 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:26.549649 2576 patch_prober.go:28] interesting pod/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": read tcp 10.132.0.2:43958->10.132.0.58:15021: read: connection reset by peer" start-of-body= Jul 14 19:52:26.550174 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:26.549748 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" podUID="1d6ccf82-87f6-4332-8838-46f449d15c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": read tcp 10.132.0.2:43958->10.132.0.58:15021: read: connection reset by peer" Jul 14 19:52:27.243612 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:27.243553 2576 generic.go:356] "Generic (PLEG): container finished" podID="1d6ccf82-87f6-4332-8838-46f449d15c52" containerID="25f1367a38602fc560f51f4362d67defc6f4e78d7b929a087d678e9ef25d7c7a" exitCode=0 Jul 14 19:52:27.243818 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:27.243637 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" event={"ID":"1d6ccf82-87f6-4332-8838-46f449d15c52","Type":"ContainerDied","Data":"25f1367a38602fc560f51f4362d67defc6f4e78d7b929a087d678e9ef25d7c7a"} Jul 14 19:52:27.244135 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:27.244119 2576 scope.go:117] "RemoveContainer" containerID="25f1367a38602fc560f51f4362d67defc6f4e78d7b929a087d678e9ef25d7c7a" Jul 14 19:52:27.245161 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:27.245131 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:52:27.245248 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:27.245198 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:52:27.245248 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:27.245231 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:52:28.250833 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:28.250796 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" event={"ID":"1d6ccf82-87f6-4332-8838-46f449d15c52","Type":"ContainerStarted","Data":"3096afe8f672d0641c69dc69997f349dfd21490d6cac7c63b3cabbedf01ecfeb"} Jul 14 19:52:28.877795 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:28.877754 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:28.878226 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:28.878199 2576 patch_prober.go:28] interesting pod/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 14 19:52:28.878405 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:28.878382 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" podUID="1d6ccf82-87f6-4332-8838-46f449d15c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 14 19:52:29.877690 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:29.877649 2576 patch_prober.go:28] interesting pod/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 14 19:52:29.878318 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:29.878282 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" podUID="1d6ccf82-87f6-4332-8838-46f449d15c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 14 19:52:30.877984 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:30.877943 2576 patch_prober.go:28] interesting pod/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 14 19:52:30.878466 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:30.878008 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" podUID="1d6ccf82-87f6-4332-8838-46f449d15c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 14 19:52:31.867065 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:31.867032 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922b98-xr9gk"] Jul 14 19:52:31.867475 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:31.867461 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="433f779c-ff71-466b-87eb-895b4bf6458e" containerName="istio-proxy" Jul 14 19:52:31.867534 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:31.867477 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="433f779c-ff71-466b-87eb-895b4bf6458e" containerName="istio-proxy" Jul 14 19:52:31.867602 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:31.867576 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="433f779c-ff71-466b-87eb-895b4bf6458e" containerName="istio-proxy" Jul 14 19:52:31.870125 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:31.870105 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922b98-xr9gk" Jul 14 19:52:31.873183 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:31.873154 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-br5f6\"" Jul 14 19:52:31.877208 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:31.877181 2576 patch_prober.go:28] interesting pod/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 14 19:52:31.877796 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:31.877756 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" podUID="1d6ccf82-87f6-4332-8838-46f449d15c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 14 19:52:31.882654 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:31.882558 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922b98-xr9gk"] Jul 14 19:52:31.964031 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:31.963997 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46009e58-41df-4454-be6f-159f1affa9e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-922b98-xr9gk\" (UID: \"46009e58-41df-4454-be6f-159f1affa9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922b98-xr9gk" Jul 14 19:52:32.064723 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:32.064695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46009e58-41df-4454-be6f-159f1affa9e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-922b98-xr9gk\" (UID: \"46009e58-41df-4454-be6f-159f1affa9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922b98-xr9gk" Jul 14 19:52:32.065328 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:32.065309 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46009e58-41df-4454-be6f-159f1affa9e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-922b98-xr9gk\" (UID: \"46009e58-41df-4454-be6f-159f1affa9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922b98-xr9gk" Jul 14 19:52:32.186076 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:32.185955 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922b98-xr9gk" Jul 14 19:52:32.406374 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:32.406335 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922b98-xr9gk"] Jul 14 19:52:32.880951 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:32.880924 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:32.881245 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:32.881213 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:32.882008 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:32.881991 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:52:33.286477 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:33.286379 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922b98-xr9gk" event={"ID":"46009e58-41df-4454-be6f-159f1affa9e5","Type":"ContainerStarted","Data":"9c5a5be33eb195debdcf8218625589b65e62cc23938cc7e94804550910cf0a1c"} Jul 14 19:52:33.286477 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:33.286447 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922b98-xr9gk" event={"ID":"46009e58-41df-4454-be6f-159f1affa9e5","Type":"ContainerStarted","Data":"20c1f6d30342bee1f207b16e789cc9ee32a1bb90f62d3faba96c86e015e1ab81"} Jul 14 19:52:33.303931 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:33.303874 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922b98-xr9gk" podStartSLOduration=2.30385671 podStartE2EDuration="2.30385671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:33.3016828 +0000 UTC m=+2506.414932669" watchObservedRunningTime="2026-07-14 19:52:33.30385671 +0000 UTC m=+2506.417106629" Jul 14 19:52:34.293044 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:34.293005 2576 generic.go:356] "Generic (PLEG): container finished" podID="46009e58-41df-4454-be6f-159f1affa9e5" containerID="9c5a5be33eb195debdcf8218625589b65e62cc23938cc7e94804550910cf0a1c" exitCode=0 Jul 14 19:52:34.293552 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:34.293086 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922b98-xr9gk" event={"ID":"46009e58-41df-4454-be6f-159f1affa9e5","Type":"ContainerDied","Data":"9c5a5be33eb195debdcf8218625589b65e62cc23938cc7e94804550910cf0a1c"} Jul 14 19:52:35.236811 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:35.236767 2576 patch_prober.go:28] interesting pod/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 19:52:35.237959 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:35.236846 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" podUID="eb68a821-ade5-40f2-8691-81a85a8e9373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 19:52:35.300775 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:35.300749 2576 generic.go:356] "Generic (PLEG): container finished" podID="eb68a821-ade5-40f2-8691-81a85a8e9373" containerID="60bdada8e90fdc8a1c34f51b8d20ec0751d892502d5a27735a956d1dcca17faf" exitCode=0 Jul 14 19:52:35.301133 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:35.300824 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" event={"ID":"eb68a821-ade5-40f2-8691-81a85a8e9373","Type":"ContainerDied","Data":"60bdada8e90fdc8a1c34f51b8d20ec0751d892502d5a27735a956d1dcca17faf"} Jul 14 19:52:35.301544 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:35.301400 2576 scope.go:117] "RemoveContainer" containerID="60bdada8e90fdc8a1c34f51b8d20ec0751d892502d5a27735a956d1dcca17faf" Jul 14 19:52:35.302421 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:35.302392 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:52:35.302511 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:35.302462 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:52:35.302511 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:35.302500 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:52:35.444632 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:35.444539 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922b98-xr9gk" Jul 14 19:52:35.500571 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:35.500471 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46009e58-41df-4454-be6f-159f1affa9e5-maas-api-service-ca\") pod \"46009e58-41df-4454-be6f-159f1affa9e5\" (UID: \"46009e58-41df-4454-be6f-159f1affa9e5\") " Jul 14 19:52:35.500806 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:35.500783 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46009e58-41df-4454-be6f-159f1affa9e5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "46009e58-41df-4454-be6f-159f1affa9e5" (UID: "46009e58-41df-4454-be6f-159f1affa9e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:52:35.500978 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:35.500954 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46009e58-41df-4454-be6f-159f1affa9e5-maas-api-service-ca\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:52:36.308567 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:36.308529 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" event={"ID":"eb68a821-ade5-40f2-8691-81a85a8e9373","Type":"ContainerStarted","Data":"d32acf9c71015b376cd36475304ee48149937d47a07b11f8c5f6cdee3b093dd3"} Jul 14 19:52:36.309866 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:36.309848 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922b98-xr9gk" Jul 14 19:52:36.309979 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:36.309887 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922b98-xr9gk" event={"ID":"46009e58-41df-4454-be6f-159f1affa9e5","Type":"ContainerDied","Data":"20c1f6d30342bee1f207b16e789cc9ee32a1bb90f62d3faba96c86e015e1ab81"} Jul 14 19:52:36.309979 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:36.309925 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20c1f6d30342bee1f207b16e789cc9ee32a1bb90f62d3faba96c86e015e1ab81" Jul 14 19:52:36.828343 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:36.828300 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:36.828572 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:36.828545 2576 patch_prober.go:28] interesting pod/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 14 19:52:36.828661 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:36.828640 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" podUID="eb68a821-ade5-40f2-8691-81a85a8e9373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 14 19:52:37.828406 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:37.828364 2576 patch_prober.go:28] interesting pod/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 14 19:52:37.829010 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:37.828449 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" podUID="eb68a821-ade5-40f2-8691-81a85a8e9373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 14 19:52:38.828556 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:38.828465 2576 patch_prober.go:28] interesting pod/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 14 19:52:38.828556 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:38.828526 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" podUID="eb68a821-ade5-40f2-8691-81a85a8e9373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 14 19:52:39.828745 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:39.828707 2576 patch_prober.go:28] interesting pod/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 14 19:52:39.829187 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:39.828797 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" podUID="eb68a821-ade5-40f2-8691-81a85a8e9373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 14 19:52:40.831288 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:40.831256 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:40.831809 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:40.831651 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:40.832237 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:40.832219 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:52:57.728133 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:57.728089 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9"] Jul 14 19:52:57.728629 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:57.728386 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" podUID="1d6ccf82-87f6-4332-8838-46f449d15c52" containerName="istio-proxy" containerID="cri-o://3096afe8f672d0641c69dc69997f349dfd21490d6cac7c63b3cabbedf01ecfeb" gracePeriod=30 Jul 14 19:52:58.147122 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:58.147077 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57e988-g6fzr"] Jul 14 19:52:58.147556 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:58.147535 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46009e58-41df-4454-be6f-159f1affa9e5" containerName="revoke-keys" Jul 14 19:52:58.147556 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:58.147552 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="46009e58-41df-4454-be6f-159f1affa9e5" containerName="revoke-keys" Jul 14 19:52:58.147745 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:58.147635 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="46009e58-41df-4454-be6f-159f1affa9e5" containerName="revoke-keys" Jul 14 19:52:58.150826 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:58.150809 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57e988-g6fzr" Jul 14 19:52:58.153755 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:58.153735 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-br5f6\"" Jul 14 19:52:58.160517 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:58.160487 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57e988-g6fzr"] Jul 14 19:52:58.223775 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:58.223739 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d926bbd3-7f24-40c2-893f-1e48d9d9bbd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57e988-g6fzr\" (UID: \"d926bbd3-7f24-40c2-893f-1e48d9d9bbd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57e988-g6fzr" Jul 14 19:52:58.324885 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:58.324842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d926bbd3-7f24-40c2-893f-1e48d9d9bbd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57e988-g6fzr\" (UID: \"d926bbd3-7f24-40c2-893f-1e48d9d9bbd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57e988-g6fzr" Jul 14 19:52:58.325511 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:58.325489 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d926bbd3-7f24-40c2-893f-1e48d9d9bbd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57e988-g6fzr\" (UID: \"d926bbd3-7f24-40c2-893f-1e48d9d9bbd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57e988-g6fzr" Jul 14 19:52:58.462605 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:58.462491 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57e988-g6fzr" Jul 14 19:52:58.606458 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:58.606427 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57e988-g6fzr"] Jul 14 19:52:58.608166 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:52:58.608138 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd926bbd3_7f24_40c2_893f_1e48d9d9bbd9.slice/crio-a0f0fe3a8f89ce85d40dba8da64d3dfd070db4fbd7f29fc16743b72c12f0a4a8 WatchSource:0}: Error finding container a0f0fe3a8f89ce85d40dba8da64d3dfd070db4fbd7f29fc16743b72c12f0a4a8: Status 404 returned error can't find the container with id a0f0fe3a8f89ce85d40dba8da64d3dfd070db4fbd7f29fc16743b72c12f0a4a8 Jul 14 19:52:59.433864 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:59.433829 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57e988-g6fzr" event={"ID":"d926bbd3-7f24-40c2-893f-1e48d9d9bbd9","Type":"ContainerStarted","Data":"d105d885096ff3c7692dab8bf2cf36f2da3c793d633462c481e2dfda4efa88fd"} Jul 14 19:52:59.434253 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:59.433883 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57e988-g6fzr" event={"ID":"d926bbd3-7f24-40c2-893f-1e48d9d9bbd9","Type":"ContainerStarted","Data":"a0f0fe3a8f89ce85d40dba8da64d3dfd070db4fbd7f29fc16743b72c12f0a4a8"} Jul 14 19:52:59.452050 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:52:59.451999 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57e988-g6fzr" podStartSLOduration=1.451980268 podStartE2EDuration="1.451980268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:59.450003292 +0000 UTC m=+2532.563253195" watchObservedRunningTime="2026-07-14 19:52:59.451980268 +0000 UTC m=+2532.565230194" Jul 14 19:53:00.440892 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:00.440854 2576 generic.go:356] "Generic (PLEG): container finished" podID="d926bbd3-7f24-40c2-893f-1e48d9d9bbd9" containerID="d105d885096ff3c7692dab8bf2cf36f2da3c793d633462c481e2dfda4efa88fd" exitCode=0 Jul 14 19:53:00.441349 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:00.440941 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57e988-g6fzr" event={"ID":"d926bbd3-7f24-40c2-893f-1e48d9d9bbd9","Type":"ContainerDied","Data":"d105d885096ff3c7692dab8bf2cf36f2da3c793d633462c481e2dfda4efa88fd"} Jul 14 19:53:01.591807 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:01.591777 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57e988-g6fzr" Jul 14 19:53:01.759282 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:01.759186 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d926bbd3-7f24-40c2-893f-1e48d9d9bbd9-maas-api-service-ca\") pod \"d926bbd3-7f24-40c2-893f-1e48d9d9bbd9\" (UID: \"d926bbd3-7f24-40c2-893f-1e48d9d9bbd9\") " Jul 14 19:53:01.759557 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:01.759533 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d926bbd3-7f24-40c2-893f-1e48d9d9bbd9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d926bbd3-7f24-40c2-893f-1e48d9d9bbd9" (UID: "d926bbd3-7f24-40c2-893f-1e48d9d9bbd9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:53:01.860424 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:01.860380 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d926bbd3-7f24-40c2-893f-1e48d9d9bbd9-maas-api-service-ca\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:02.456216 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:02.456180 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57e988-g6fzr" Jul 14 19:53:02.456397 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:02.456186 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57e988-g6fzr" event={"ID":"d926bbd3-7f24-40c2-893f-1e48d9d9bbd9","Type":"ContainerDied","Data":"a0f0fe3a8f89ce85d40dba8da64d3dfd070db4fbd7f29fc16743b72c12f0a4a8"} Jul 14 19:53:02.456397 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:02.456306 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0f0fe3a8f89ce85d40dba8da64d3dfd070db4fbd7f29fc16743b72c12f0a4a8" Jul 14 19:53:02.883012 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:02.882980 2576 patch_prober.go:28] interesting pod/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 14 19:53:02.883440 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:02.883036 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" podUID="1d6ccf82-87f6-4332-8838-46f449d15c52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 14 19:53:02.991037 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:02.990968 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:53:03.170856 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.170824 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-envoy\") pod \"1d6ccf82-87f6-4332-8838-46f449d15c52\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " Jul 14 19:53:03.171049 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.170883 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jt2wt\" (UniqueName: \"kubernetes.io/projected/1d6ccf82-87f6-4332-8838-46f449d15c52-kube-api-access-jt2wt\") pod \"1d6ccf82-87f6-4332-8838-46f449d15c52\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " Jul 14 19:53:03.171049 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.170927 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-token\") pod \"1d6ccf82-87f6-4332-8838-46f449d15c52\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " Jul 14 19:53:03.171049 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.170949 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-credential-socket\") pod \"1d6ccf82-87f6-4332-8838-46f449d15c52\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " Jul 14 19:53:03.171049 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.170983 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-podinfo\") pod \"1d6ccf82-87f6-4332-8838-46f449d15c52\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " Jul 14 19:53:03.171049 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.171020 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-workload-certs\") pod \"1d6ccf82-87f6-4332-8838-46f449d15c52\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " Jul 14 19:53:03.171326 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.171090 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-data\") pod \"1d6ccf82-87f6-4332-8838-46f449d15c52\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " Jul 14 19:53:03.171326 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.171121 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-workload-socket\") pod \"1d6ccf82-87f6-4332-8838-46f449d15c52\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " Jul 14 19:53:03.171326 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.171179 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ccf82-87f6-4332-8838-46f449d15c52-istiod-ca-cert\") pod \"1d6ccf82-87f6-4332-8838-46f449d15c52\" (UID: \"1d6ccf82-87f6-4332-8838-46f449d15c52\") " Jul 14 19:53:03.171467 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.171321 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1d6ccf82-87f6-4332-8838-46f449d15c52" (UID: "1d6ccf82-87f6-4332-8838-46f449d15c52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:03.171467 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.171448 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1d6ccf82-87f6-4332-8838-46f449d15c52" (UID: "1d6ccf82-87f6-4332-8838-46f449d15c52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:03.171786 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.171668 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-credential-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:03.171786 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.171714 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-workload-certs\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:03.171786 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.171688 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d6ccf82-87f6-4332-8838-46f449d15c52-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1d6ccf82-87f6-4332-8838-46f449d15c52" (UID: "1d6ccf82-87f6-4332-8838-46f449d15c52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:53:03.172053 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.171782 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1d6ccf82-87f6-4332-8838-46f449d15c52" (UID: "1d6ccf82-87f6-4332-8838-46f449d15c52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:03.172247 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.172226 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-data" (OuterVolumeSpecName: "istio-data") pod "1d6ccf82-87f6-4332-8838-46f449d15c52" (UID: "1d6ccf82-87f6-4332-8838-46f449d15c52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:03.173296 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.173268 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6ccf82-87f6-4332-8838-46f449d15c52-kube-api-access-jt2wt" (OuterVolumeSpecName: "kube-api-access-jt2wt") pod "1d6ccf82-87f6-4332-8838-46f449d15c52" (UID: "1d6ccf82-87f6-4332-8838-46f449d15c52"). InnerVolumeSpecName "kube-api-access-jt2wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:53:03.173387 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.173279 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-token" (OuterVolumeSpecName: "istio-token") pod "1d6ccf82-87f6-4332-8838-46f449d15c52" (UID: "1d6ccf82-87f6-4332-8838-46f449d15c52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:53:03.173575 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.173550 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1d6ccf82-87f6-4332-8838-46f449d15c52" (UID: "1d6ccf82-87f6-4332-8838-46f449d15c52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:03.173721 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.173697 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1d6ccf82-87f6-4332-8838-46f449d15c52" (UID: "1d6ccf82-87f6-4332-8838-46f449d15c52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 19:53:03.272912 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.272817 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-data\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:03.272912 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.272850 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-workload-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:03.272912 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.272859 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6ccf82-87f6-4332-8838-46f449d15c52-istiod-ca-cert\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:03.272912 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.272871 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-envoy\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:03.272912 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.272879 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt2wt\" (UniqueName: \"kubernetes.io/projected/1d6ccf82-87f6-4332-8838-46f449d15c52-kube-api-access-jt2wt\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:03.272912 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.272889 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-token\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:03.272912 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.272897 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6ccf82-87f6-4332-8838-46f449d15c52-istio-podinfo\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:03.463070 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.463037 2576 generic.go:356] "Generic (PLEG): container finished" podID="1d6ccf82-87f6-4332-8838-46f449d15c52" containerID="3096afe8f672d0641c69dc69997f349dfd21490d6cac7c63b3cabbedf01ecfeb" exitCode=0 Jul 14 19:53:03.463263 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.463094 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" Jul 14 19:53:03.463263 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.463127 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" event={"ID":"1d6ccf82-87f6-4332-8838-46f449d15c52","Type":"ContainerDied","Data":"3096afe8f672d0641c69dc69997f349dfd21490d6cac7c63b3cabbedf01ecfeb"} Jul 14 19:53:03.463263 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.463186 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9" event={"ID":"1d6ccf82-87f6-4332-8838-46f449d15c52","Type":"ContainerDied","Data":"6c517d9c751181c58fbfae007d426aca3b4d7326bce9ccbed14bf9174141acff"} Jul 14 19:53:03.463263 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.463209 2576 scope.go:117] "RemoveContainer" containerID="3096afe8f672d0641c69dc69997f349dfd21490d6cac7c63b3cabbedf01ecfeb" Jul 14 19:53:03.493840 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.473884 2576 scope.go:117] "RemoveContainer" containerID="25f1367a38602fc560f51f4362d67defc6f4e78d7b929a087d678e9ef25d7c7a" Jul 14 19:53:03.493840 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.490616 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9"] Jul 14 19:53:03.495228 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.495186 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-922b98-openshift-default-69cd888d8c-t2rp9"] Jul 14 19:53:03.503900 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.503866 2576 scope.go:117] "RemoveContainer" containerID="3096afe8f672d0641c69dc69997f349dfd21490d6cac7c63b3cabbedf01ecfeb" Jul 14 19:53:03.504236 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:53:03.504209 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3096afe8f672d0641c69dc69997f349dfd21490d6cac7c63b3cabbedf01ecfeb\": container with ID starting with 3096afe8f672d0641c69dc69997f349dfd21490d6cac7c63b3cabbedf01ecfeb not found: ID does not exist" containerID="3096afe8f672d0641c69dc69997f349dfd21490d6cac7c63b3cabbedf01ecfeb" Jul 14 19:53:03.504345 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.504243 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3096afe8f672d0641c69dc69997f349dfd21490d6cac7c63b3cabbedf01ecfeb"} err="failed to get container status \"3096afe8f672d0641c69dc69997f349dfd21490d6cac7c63b3cabbedf01ecfeb\": rpc error: code = NotFound desc = could not find container \"3096afe8f672d0641c69dc69997f349dfd21490d6cac7c63b3cabbedf01ecfeb\": container with ID starting with 3096afe8f672d0641c69dc69997f349dfd21490d6cac7c63b3cabbedf01ecfeb not found: ID does not exist" Jul 14 19:53:03.504345 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.504263 2576 scope.go:117] "RemoveContainer" containerID="25f1367a38602fc560f51f4362d67defc6f4e78d7b929a087d678e9ef25d7c7a" Jul 14 19:53:03.504499 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:53:03.504486 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f1367a38602fc560f51f4362d67defc6f4e78d7b929a087d678e9ef25d7c7a\": container with ID starting with 25f1367a38602fc560f51f4362d67defc6f4e78d7b929a087d678e9ef25d7c7a not found: ID does not exist" containerID="25f1367a38602fc560f51f4362d67defc6f4e78d7b929a087d678e9ef25d7c7a" Jul 14 19:53:03.504543 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.504505 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f1367a38602fc560f51f4362d67defc6f4e78d7b929a087d678e9ef25d7c7a"} err="failed to get container status \"25f1367a38602fc560f51f4362d67defc6f4e78d7b929a087d678e9ef25d7c7a\": rpc error: code = NotFound desc = could not find container \"25f1367a38602fc560f51f4362d67defc6f4e78d7b929a087d678e9ef25d7c7a\": container with ID starting with 25f1367a38602fc560f51f4362d67defc6f4e78d7b929a087d678e9ef25d7c7a not found: ID does not exist" Jul 14 19:53:03.515282 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:03.515251 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d6ccf82-87f6-4332-8838-46f449d15c52" path="/var/lib/kubelet/pods/1d6ccf82-87f6-4332-8838-46f449d15c52/volumes" Jul 14 19:53:23.988768 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:23.988724 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt"] Jul 14 19:53:23.989395 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:23.988961 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" podUID="eb68a821-ade5-40f2-8691-81a85a8e9373" containerName="istio-proxy" containerID="cri-o://d32acf9c71015b376cd36475304ee48149937d47a07b11f8c5f6cdee3b093dd3" gracePeriod=30 Jul 14 19:53:24.923422 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.923385 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c"] Jul 14 19:53:24.923906 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.923885 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d6ccf82-87f6-4332-8838-46f449d15c52" containerName="istio-proxy" Jul 14 19:53:24.924014 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.923908 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d6ccf82-87f6-4332-8838-46f449d15c52" containerName="istio-proxy" Jul 14 19:53:24.924014 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.923925 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d6ccf82-87f6-4332-8838-46f449d15c52" containerName="istio-proxy" Jul 14 19:53:24.924014 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.923933 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d6ccf82-87f6-4332-8838-46f449d15c52" containerName="istio-proxy" Jul 14 19:53:24.924014 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.923950 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d926bbd3-7f24-40c2-893f-1e48d9d9bbd9" containerName="revoke-keys" Jul 14 19:53:24.924014 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.923958 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d926bbd3-7f24-40c2-893f-1e48d9d9bbd9" containerName="revoke-keys" Jul 14 19:53:24.924334 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.924104 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d926bbd3-7f24-40c2-893f-1e48d9d9bbd9" containerName="revoke-keys" Jul 14 19:53:24.924334 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.924122 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d6ccf82-87f6-4332-8838-46f449d15c52" containerName="istio-proxy" Jul 14 19:53:24.927486 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.927464 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:24.930482 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.930459 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-70659c-openshift-default-dockercfg-zl48l\"" Jul 14 19:53:24.940742 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.940714 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c"] Jul 14 19:53:24.991084 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.991040 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-istio-envoy\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:24.991497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.991099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zs8h\" (UniqueName: \"kubernetes.io/projected/a8359799-f862-404f-bfb6-a3a018233b9a-kube-api-access-2zs8h\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:24.991497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.991127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8359799-f862-404f-bfb6-a3a018233b9a-istio-podinfo\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:24.991497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.991186 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8359799-f862-404f-bfb6-a3a018233b9a-istiod-ca-cert\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:24.991497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.991218 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-credential-socket\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:24.991497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.991236 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-workload-socket\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:24.991497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.991355 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-workload-certs\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:24.991497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.991377 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-istio-data\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:24.991497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:24.991398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8359799-f862-404f-bfb6-a3a018233b9a-istio-token\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.092925 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.092874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-istio-envoy\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.092925 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.092924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zs8h\" (UniqueName: \"kubernetes.io/projected/a8359799-f862-404f-bfb6-a3a018233b9a-kube-api-access-2zs8h\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.093172 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.092959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8359799-f862-404f-bfb6-a3a018233b9a-istio-podinfo\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.093172 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.093000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8359799-f862-404f-bfb6-a3a018233b9a-istiod-ca-cert\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.093172 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.093046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-credential-socket\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.093172 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.093078 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-workload-socket\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.093172 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.093169 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-workload-certs\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.093426 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.093192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-istio-data\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.093482 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.093423 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8359799-f862-404f-bfb6-a3a018233b9a-istio-token\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.094024 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.093883 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-credential-socket\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.094024 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.093996 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8359799-f862-404f-bfb6-a3a018233b9a-istiod-ca-cert\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.094227 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.094106 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-workload-socket\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.094227 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.094115 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-workload-certs\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.094227 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.094175 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-istio-data\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.095736 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.095706 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-istio-envoy\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.095931 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.095912 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8359799-f862-404f-bfb6-a3a018233b9a-istio-podinfo\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.104138 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.104108 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8359799-f862-404f-bfb6-a3a018233b9a-istio-token\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.105022 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.104970 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zs8h\" (UniqueName: \"kubernetes.io/projected/a8359799-f862-404f-bfb6-a3a018233b9a-kube-api-access-2zs8h\") pod \"e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.237901 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.237810 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:25.393382 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.393353 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c"] Jul 14 19:53:25.396263 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:53:25.396233 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8359799_f862_404f_bfb6_a3a018233b9a.slice/crio-50e67c6168abb7fac51ff979f7995ae7f7a868c7edb40d30d3190dc19a8d8fbf WatchSource:0}: Error finding container 50e67c6168abb7fac51ff979f7995ae7f7a868c7edb40d30d3190dc19a8d8fbf: Status 404 returned error can't find the container with id 50e67c6168abb7fac51ff979f7995ae7f7a868c7edb40d30d3190dc19a8d8fbf Jul 14 19:53:25.398358 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.398319 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:53:25.398428 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.398393 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:53:25.398467 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.398427 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:53:25.584753 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.584721 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" event={"ID":"a8359799-f862-404f-bfb6-a3a018233b9a","Type":"ContainerStarted","Data":"5be89fb78d3527ee4b4b49558f00c7b8603f0da2aa65a509ead4fca0647a588b"} Jul 14 19:53:25.584927 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.584773 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" event={"ID":"a8359799-f862-404f-bfb6-a3a018233b9a","Type":"ContainerStarted","Data":"50e67c6168abb7fac51ff979f7995ae7f7a868c7edb40d30d3190dc19a8d8fbf"} Jul 14 19:53:25.607192 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.607143 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" podStartSLOduration=1.60712513 podStartE2EDuration="1.60712513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:53:25.605275644 +0000 UTC m=+2558.718525515" watchObservedRunningTime="2026-07-14 19:53:25.60712513 +0000 UTC m=+2558.720375001" Jul 14 19:53:25.833898 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.833866 2576 patch_prober.go:28] interesting pod/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 19:53:25.834036 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:25.833924 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" podUID="eb68a821-ade5-40f2-8691-81a85a8e9373" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 19:53:26.238819 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:26.238780 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:26.243691 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:26.243664 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:26.590659 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:26.590625 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:26.591792 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:26.591758 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:29.243036 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.243001 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:53:29.333048 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.333012 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-token\") pod \"eb68a821-ade5-40f2-8691-81a85a8e9373\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " Jul 14 19:53:29.333223 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.333086 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-data\") pod \"eb68a821-ade5-40f2-8691-81a85a8e9373\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " Jul 14 19:53:29.333223 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.333128 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-envoy\") pod \"eb68a821-ade5-40f2-8691-81a85a8e9373\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " Jul 14 19:53:29.333223 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.333148 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb68a821-ade5-40f2-8691-81a85a8e9373-istiod-ca-cert\") pod \"eb68a821-ade5-40f2-8691-81a85a8e9373\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " Jul 14 19:53:29.333223 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.333178 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-credential-socket\") pod \"eb68a821-ade5-40f2-8691-81a85a8e9373\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " Jul 14 19:53:29.333406 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.333346 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-workload-certs\") pod \"eb68a821-ade5-40f2-8691-81a85a8e9373\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " Jul 14 19:53:29.333406 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.333399 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98vjt\" (UniqueName: \"kubernetes.io/projected/eb68a821-ade5-40f2-8691-81a85a8e9373-kube-api-access-98vjt\") pod \"eb68a821-ade5-40f2-8691-81a85a8e9373\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " Jul 14 19:53:29.333508 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.333432 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-podinfo\") pod \"eb68a821-ade5-40f2-8691-81a85a8e9373\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " Jul 14 19:53:29.333508 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.333465 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-workload-socket\") pod \"eb68a821-ade5-40f2-8691-81a85a8e9373\" (UID: \"eb68a821-ade5-40f2-8691-81a85a8e9373\") " Jul 14 19:53:29.333508 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.333484 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eb68a821-ade5-40f2-8691-81a85a8e9373" (UID: "eb68a821-ade5-40f2-8691-81a85a8e9373"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:29.333699 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.333653 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eb68a821-ade5-40f2-8691-81a85a8e9373" (UID: "eb68a821-ade5-40f2-8691-81a85a8e9373"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:29.333699 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.333665 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb68a821-ade5-40f2-8691-81a85a8e9373-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eb68a821-ade5-40f2-8691-81a85a8e9373" (UID: "eb68a821-ade5-40f2-8691-81a85a8e9373"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:53:29.333932 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.333881 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eb68a821-ade5-40f2-8691-81a85a8e9373" (UID: "eb68a821-ade5-40f2-8691-81a85a8e9373"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:29.334015 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.333934 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb68a821-ade5-40f2-8691-81a85a8e9373-istiod-ca-cert\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:29.334015 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.333956 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-credential-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:29.334015 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.333972 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-workload-certs\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:29.334121 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.334032 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-data" (OuterVolumeSpecName: "istio-data") pod "eb68a821-ade5-40f2-8691-81a85a8e9373" (UID: "eb68a821-ade5-40f2-8691-81a85a8e9373"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:29.335867 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.335843 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eb68a821-ade5-40f2-8691-81a85a8e9373" (UID: "eb68a821-ade5-40f2-8691-81a85a8e9373"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:29.335961 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.335860 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-token" (OuterVolumeSpecName: "istio-token") pod "eb68a821-ade5-40f2-8691-81a85a8e9373" (UID: "eb68a821-ade5-40f2-8691-81a85a8e9373"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:53:29.335961 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.335930 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eb68a821-ade5-40f2-8691-81a85a8e9373" (UID: "eb68a821-ade5-40f2-8691-81a85a8e9373"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 19:53:29.336034 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.336011 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb68a821-ade5-40f2-8691-81a85a8e9373-kube-api-access-98vjt" (OuterVolumeSpecName: "kube-api-access-98vjt") pod "eb68a821-ade5-40f2-8691-81a85a8e9373" (UID: "eb68a821-ade5-40f2-8691-81a85a8e9373"). InnerVolumeSpecName "kube-api-access-98vjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:53:29.435548 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.435498 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-token\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:29.435548 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.435544 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-data\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:29.435548 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.435554 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-envoy\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:29.435795 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.435564 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98vjt\" (UniqueName: \"kubernetes.io/projected/eb68a821-ade5-40f2-8691-81a85a8e9373-kube-api-access-98vjt\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:29.435795 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.435574 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb68a821-ade5-40f2-8691-81a85a8e9373-istio-podinfo\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:29.435795 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.435604 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb68a821-ade5-40f2-8691-81a85a8e9373-workload-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:53:29.607616 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.607503 2576 generic.go:356] "Generic (PLEG): container finished" podID="eb68a821-ade5-40f2-8691-81a85a8e9373" containerID="d32acf9c71015b376cd36475304ee48149937d47a07b11f8c5f6cdee3b093dd3" exitCode=0 Jul 14 19:53:29.607616 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.607561 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" Jul 14 19:53:29.607894 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.607617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" event={"ID":"eb68a821-ade5-40f2-8691-81a85a8e9373","Type":"ContainerDied","Data":"d32acf9c71015b376cd36475304ee48149937d47a07b11f8c5f6cdee3b093dd3"} Jul 14 19:53:29.607894 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.607674 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt" event={"ID":"eb68a821-ade5-40f2-8691-81a85a8e9373","Type":"ContainerDied","Data":"4964c2aad7bbed29a1b88fc3d533396b73d4e9896b3afb85c6cad1c5988d3983"} Jul 14 19:53:29.607894 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.607699 2576 scope.go:117] "RemoveContainer" containerID="d32acf9c71015b376cd36475304ee48149937d47a07b11f8c5f6cdee3b093dd3" Jul 14 19:53:29.617346 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.617326 2576 scope.go:117] "RemoveContainer" containerID="60bdada8e90fdc8a1c34f51b8d20ec0751d892502d5a27735a956d1dcca17faf" Jul 14 19:53:29.626405 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.626374 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt"] Jul 14 19:53:29.626672 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.626656 2576 scope.go:117] "RemoveContainer" containerID="d32acf9c71015b376cd36475304ee48149937d47a07b11f8c5f6cdee3b093dd3" Jul 14 19:53:29.627041 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:53:29.627014 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d32acf9c71015b376cd36475304ee48149937d47a07b11f8c5f6cdee3b093dd3\": container with ID starting with d32acf9c71015b376cd36475304ee48149937d47a07b11f8c5f6cdee3b093dd3 not found: ID does not exist" containerID="d32acf9c71015b376cd36475304ee48149937d47a07b11f8c5f6cdee3b093dd3" Jul 14 19:53:29.627114 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.627051 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d32acf9c71015b376cd36475304ee48149937d47a07b11f8c5f6cdee3b093dd3"} err="failed to get container status \"d32acf9c71015b376cd36475304ee48149937d47a07b11f8c5f6cdee3b093dd3\": rpc error: code = NotFound desc = could not find container \"d32acf9c71015b376cd36475304ee48149937d47a07b11f8c5f6cdee3b093dd3\": container with ID starting with d32acf9c71015b376cd36475304ee48149937d47a07b11f8c5f6cdee3b093dd3 not found: ID does not exist" Jul 14 19:53:29.627114 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.627070 2576 scope.go:117] "RemoveContainer" containerID="60bdada8e90fdc8a1c34f51b8d20ec0751d892502d5a27735a956d1dcca17faf" Jul 14 19:53:29.627331 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:53:29.627313 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60bdada8e90fdc8a1c34f51b8d20ec0751d892502d5a27735a956d1dcca17faf\": container with ID starting with 60bdada8e90fdc8a1c34f51b8d20ec0751d892502d5a27735a956d1dcca17faf not found: ID does not exist" containerID="60bdada8e90fdc8a1c34f51b8d20ec0751d892502d5a27735a956d1dcca17faf" Jul 14 19:53:29.627375 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.627337 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60bdada8e90fdc8a1c34f51b8d20ec0751d892502d5a27735a956d1dcca17faf"} err="failed to get container status \"60bdada8e90fdc8a1c34f51b8d20ec0751d892502d5a27735a956d1dcca17faf\": rpc error: code = NotFound desc = could not find container \"60bdada8e90fdc8a1c34f51b8d20ec0751d892502d5a27735a956d1dcca17faf\": container with ID starting with 60bdada8e90fdc8a1c34f51b8d20ec0751d892502d5a27735a956d1dcca17faf not found: ID does not exist" Jul 14 19:53:29.631852 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:29.631827 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-57e988-openshift-default-548cf5c769-6bfnt"] Jul 14 19:53:31.516167 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:31.516138 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb68a821-ade5-40f2-8691-81a85a8e9373" path="/var/lib/kubelet/pods/eb68a821-ade5-40f2-8691-81a85a8e9373/volumes" Jul 14 19:53:32.755858 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.755814 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f"] Jul 14 19:53:32.756323 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.756250 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb68a821-ade5-40f2-8691-81a85a8e9373" containerName="istio-proxy" Jul 14 19:53:32.756323 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.756263 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb68a821-ade5-40f2-8691-81a85a8e9373" containerName="istio-proxy" Jul 14 19:53:32.756323 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.756271 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb68a821-ade5-40f2-8691-81a85a8e9373" containerName="istio-proxy" Jul 14 19:53:32.756323 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.756276 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb68a821-ade5-40f2-8691-81a85a8e9373" containerName="istio-proxy" Jul 14 19:53:32.756486 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.756349 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb68a821-ade5-40f2-8691-81a85a8e9373" containerName="istio-proxy" Jul 14 19:53:32.756486 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.756364 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d6ccf82-87f6-4332-8838-46f449d15c52" containerName="istio-proxy" Jul 14 19:53:32.761030 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.761009 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.763680 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.763658 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-356b53-openshift-default-dockercfg-lk4bv\"" Jul 14 19:53:32.772751 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.772718 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f"] Jul 14 19:53:32.873481 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.873442 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-envoy\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.873850 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.873808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-workload-certs\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.873980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.873858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxfht\" (UniqueName: \"kubernetes.io/projected/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-kube-api-access-wxfht\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.873980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.873935 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-credential-socket\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.873980 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.873958 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-token\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.874205 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.874010 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-data\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.874205 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.874074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istiod-ca-cert\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.874205 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.874118 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-podinfo\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.874205 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.874164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-workload-socket\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.975406 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.975365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-envoy\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.975633 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.975430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-workload-certs\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.975633 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.975464 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxfht\" (UniqueName: \"kubernetes.io/projected/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-kube-api-access-wxfht\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.975633 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.975529 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-credential-socket\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.975633 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.975557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-token\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.975633 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.975597 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-data\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.975633 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.975627 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istiod-ca-cert\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.975964 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.975666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-podinfo\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.975964 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.975737 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-workload-socket\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.975964 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.975907 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-workload-certs\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.976122 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.976107 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-workload-socket\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.976594 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.976554 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istiod-ca-cert\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.976850 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.976829 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-credential-socket\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.976927 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.976879 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-data\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.978084 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.978054 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-envoy\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.978548 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.978518 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-podinfo\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.985161 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.985133 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-token\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:32.985450 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:32.985428 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxfht\" (UniqueName: \"kubernetes.io/projected/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-kube-api-access-wxfht\") pod \"e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:33.072863 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:33.072831 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:33.229285 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:33.229076 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f"] Jul 14 19:53:33.234832 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:53:33.234676 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092f7b9b_bc8d_426c_b828_1c8cdbb9ea1b.slice/crio-b355b7860c2250fbf2bc300181e335053d26508f8b729c5906dfaa29ab136681 WatchSource:0}: Error finding container b355b7860c2250fbf2bc300181e335053d26508f8b729c5906dfaa29ab136681: Status 404 returned error can't find the container with id b355b7860c2250fbf2bc300181e335053d26508f8b729c5906dfaa29ab136681 Jul 14 19:53:33.237155 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:33.237116 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:53:33.237262 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:33.237186 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:53:33.237262 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:33.237214 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:53:33.631857 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:33.631815 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" event={"ID":"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b","Type":"ContainerStarted","Data":"294a0a5baf585f0971554a3bf763e33e8e6d57cca2b67a3f4c78f1884a48fe5c"} Jul 14 19:53:33.632048 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:33.631884 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" event={"ID":"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b","Type":"ContainerStarted","Data":"b355b7860c2250fbf2bc300181e335053d26508f8b729c5906dfaa29ab136681"} Jul 14 19:53:33.654293 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:33.654236 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" podStartSLOduration=1.654218091 podStartE2EDuration="1.654218091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:53:33.652917206 +0000 UTC m=+2566.766167078" watchObservedRunningTime="2026-07-14 19:53:33.654218091 +0000 UTC m=+2566.767467960" Jul 14 19:53:34.073527 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:34.073484 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:34.078827 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:34.078802 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:34.637400 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:34.637369 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:34.638454 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:34.638435 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:41.848684 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:41.848635 2576 patch_prober.go:28] interesting pod/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": read tcp 10.132.0.2:46774->10.132.0.62:15021: read: connection reset by peer" start-of-body= Jul 14 19:53:41.849124 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:41.848737 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" podUID="a8359799-f862-404f-bfb6-a3a018233b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": read tcp 10.132.0.2:46774->10.132.0.62:15021: read: connection reset by peer" Jul 14 19:53:42.695078 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:42.695033 2576 generic.go:356] "Generic (PLEG): container finished" podID="a8359799-f862-404f-bfb6-a3a018233b9a" containerID="5be89fb78d3527ee4b4b49558f00c7b8603f0da2aa65a509ead4fca0647a588b" exitCode=0 Jul 14 19:53:42.695282 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:42.695123 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" event={"ID":"a8359799-f862-404f-bfb6-a3a018233b9a","Type":"ContainerDied","Data":"5be89fb78d3527ee4b4b49558f00c7b8603f0da2aa65a509ead4fca0647a588b"} Jul 14 19:53:42.695611 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:42.695570 2576 scope.go:117] "RemoveContainer" containerID="5be89fb78d3527ee4b4b49558f00c7b8603f0da2aa65a509ead4fca0647a588b" Jul 14 19:53:42.697208 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:42.697163 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:53:42.697310 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:42.697260 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:53:42.697376 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:42.697312 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:53:43.706034 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:43.705988 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" event={"ID":"a8359799-f862-404f-bfb6-a3a018233b9a","Type":"ContainerStarted","Data":"7e50b1e3f0474535c111062884c98ba0d083c3fe34ac99e57218429ab3bd3b5c"} Jul 14 19:53:44.238931 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:44.238898 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:44.239122 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:44.239004 2576 patch_prober.go:28] interesting pod/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 14 19:53:44.239122 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:44.239045 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" podUID="a8359799-f862-404f-bfb6-a3a018233b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 14 19:53:45.238754 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:45.238714 2576 patch_prober.go:28] interesting pod/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 14 19:53:45.239172 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:45.238787 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" podUID="a8359799-f862-404f-bfb6-a3a018233b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 14 19:53:46.238388 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:46.238354 2576 patch_prober.go:28] interesting pod/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 14 19:53:46.238569 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:46.238408 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" podUID="a8359799-f862-404f-bfb6-a3a018233b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 14 19:53:47.238967 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:47.238925 2576 patch_prober.go:28] interesting pod/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 14 19:53:47.239369 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:47.238999 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" podUID="a8359799-f862-404f-bfb6-a3a018233b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 14 19:53:48.242344 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:48.242311 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:48.242747 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:48.242698 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:48.243541 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:48.243522 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:53:48.736971 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:48.736890 2576 generic.go:356] "Generic (PLEG): container finished" podID="092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" containerID="294a0a5baf585f0971554a3bf763e33e8e6d57cca2b67a3f4c78f1884a48fe5c" exitCode=0 Jul 14 19:53:48.736971 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:48.736962 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" event={"ID":"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b","Type":"ContainerDied","Data":"294a0a5baf585f0971554a3bf763e33e8e6d57cca2b67a3f4c78f1884a48fe5c"} Jul 14 19:53:48.737456 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:48.737441 2576 scope.go:117] "RemoveContainer" containerID="294a0a5baf585f0971554a3bf763e33e8e6d57cca2b67a3f4c78f1884a48fe5c" Jul 14 19:53:48.738497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:48.738470 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:53:48.738556 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:48.738535 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:53:48.738621 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:48.738565 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:53:49.637889 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:49.637845 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:49.744019 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:49.743979 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" event={"ID":"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b","Type":"ContainerStarted","Data":"bd23f17e22f1c9c0c2eed5bc1e24f54184f2b84148183e3978739e2947a099e9"} Jul 14 19:53:50.073678 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:50.073639 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:50.075165 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:50.074813 2576 patch_prober.go:28] interesting pod/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 14 19:53:50.075165 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:50.074878 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" podUID="092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 14 19:53:51.073764 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:51.073722 2576 patch_prober.go:28] interesting pod/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 14 19:53:51.074291 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:51.073794 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" podUID="092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 14 19:53:52.074306 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:52.074272 2576 patch_prober.go:28] interesting pod/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 14 19:53:52.074926 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:52.074893 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" podUID="092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 14 19:53:53.074735 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:53.074252 2576 patch_prober.go:28] interesting pod/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 14 19:53:53.074735 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:53.074317 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" podUID="092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 14 19:53:54.077222 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:54.077185 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:54.077701 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:54.077607 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:53:54.078314 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:53:54.078296 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 19:55:47.626928 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:55:47.626846 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:55:47.644355 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:55:47.644327 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 19:58:59.561347 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:58:59.561298 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7431-140405581687088"] Jul 14 19:58:59.563778 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:58:59.561953 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb68a821-ade5-40f2-8691-81a85a8e9373" containerName="istio-proxy" Jul 14 19:58:59.564770 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:58:59.564751 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581687088" Jul 14 19:58:59.568117 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:58:59.568096 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zrwrl\"" Jul 14 19:58:59.569986 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:58:59.569962 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140405581687088"] Jul 14 19:58:59.658866 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:58:59.658830 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn96w\" (UniqueName: \"kubernetes.io/projected/f9cf3a8c-1c6d-46da-895e-00511e8b3bf1-kube-api-access-xn96w\") pod \"test-curl-7431-140405581687088\" (UID: \"f9cf3a8c-1c6d-46da-895e-00511e8b3bf1\") " pod="opendatahub/test-curl-7431-140405581687088" Jul 14 19:58:59.759806 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:58:59.759764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn96w\" (UniqueName: \"kubernetes.io/projected/f9cf3a8c-1c6d-46da-895e-00511e8b3bf1-kube-api-access-xn96w\") pod \"test-curl-7431-140405581687088\" (UID: \"f9cf3a8c-1c6d-46da-895e-00511e8b3bf1\") " pod="opendatahub/test-curl-7431-140405581687088" Jul 14 19:58:59.768419 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:58:59.768372 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn96w\" (UniqueName: \"kubernetes.io/projected/f9cf3a8c-1c6d-46da-895e-00511e8b3bf1-kube-api-access-xn96w\") pod \"test-curl-7431-140405581687088\" (UID: \"f9cf3a8c-1c6d-46da-895e-00511e8b3bf1\") " pod="opendatahub/test-curl-7431-140405581687088" Jul 14 19:58:59.875472 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:58:59.875436 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581687088" Jul 14 19:59:00.020273 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:00.020240 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140405581687088"] Jul 14 19:59:00.022399 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:59:00.022371 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9cf3a8c_1c6d_46da_895e_00511e8b3bf1.slice/crio-f45c2d6c5f87d2f77e3a5d1b247171718604e07176a2fa438ba79a9e5659f92f WatchSource:0}: Error finding container f45c2d6c5f87d2f77e3a5d1b247171718604e07176a2fa438ba79a9e5659f92f: Status 404 returned error can't find the container with id f45c2d6c5f87d2f77e3a5d1b247171718604e07176a2fa438ba79a9e5659f92f Jul 14 19:59:00.024419 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:00.024399 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:59:00.427110 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:00.427061 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140405581687088" event={"ID":"f9cf3a8c-1c6d-46da-895e-00511e8b3bf1","Type":"ContainerStarted","Data":"f45c2d6c5f87d2f77e3a5d1b247171718604e07176a2fa438ba79a9e5659f92f"} Jul 14 19:59:01.433571 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:01.433534 2576 generic.go:356] "Generic (PLEG): container finished" podID="f9cf3a8c-1c6d-46da-895e-00511e8b3bf1" containerID="85a6096d8ae35d309a2f8210a8b4c17e0ae55f5e21749cfc18ae99a2f9005d93" exitCode=0 Jul 14 19:59:01.433984 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:01.433616 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140405581687088" event={"ID":"f9cf3a8c-1c6d-46da-895e-00511e8b3bf1","Type":"ContainerDied","Data":"85a6096d8ae35d309a2f8210a8b4c17e0ae55f5e21749cfc18ae99a2f9005d93"} Jul 14 19:59:02.574360 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:02.574336 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581687088" Jul 14 19:59:02.607816 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:02.607783 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7431-140405581687088_f9cf3a8c-1c6d-46da-895e-00511e8b3bf1/test-curl-7431-140405581687088/0.log" Jul 14 19:59:02.631761 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:02.631727 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7431-140405581687088"] Jul 14 19:59:02.633856 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:02.633834 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7431-140405581687088"] Jul 14 19:59:02.692808 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:02.692714 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xn96w\" (UniqueName: \"kubernetes.io/projected/f9cf3a8c-1c6d-46da-895e-00511e8b3bf1-kube-api-access-xn96w\") pod \"f9cf3a8c-1c6d-46da-895e-00511e8b3bf1\" (UID: \"f9cf3a8c-1c6d-46da-895e-00511e8b3bf1\") " Jul 14 19:59:02.694977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:02.694949 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9cf3a8c-1c6d-46da-895e-00511e8b3bf1-kube-api-access-xn96w" (OuterVolumeSpecName: "kube-api-access-xn96w") pod "f9cf3a8c-1c6d-46da-895e-00511e8b3bf1" (UID: "f9cf3a8c-1c6d-46da-895e-00511e8b3bf1"). InnerVolumeSpecName "kube-api-access-xn96w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:59:02.794207 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:02.794168 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn96w\" (UniqueName: \"kubernetes.io/projected/f9cf3a8c-1c6d-46da-895e-00511e8b3bf1-kube-api-access-xn96w\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:59:03.446290 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:03.446254 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581687088" Jul 14 19:59:03.446470 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:03.446294 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f45c2d6c5f87d2f77e3a5d1b247171718604e07176a2fa438ba79a9e5659f92f" Jul 14 19:59:03.456302 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:59:03.456276 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7431-140405581687088\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="f9cf3a8c-1c6d-46da-895e-00511e8b3bf1" pod="opendatahub/test-curl-7431-140405581687088" Jul 14 19:59:03.521974 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:03.521941 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9cf3a8c-1c6d-46da-895e-00511e8b3bf1" path="/var/lib/kubelet/pods/f9cf3a8c-1c6d-46da-895e-00511e8b3bf1/volumes" Jul 14 19:59:05.898438 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:05.898400 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7431-140405581852720"] Jul 14 19:59:05.898849 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:05.898827 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9cf3a8c-1c6d-46da-895e-00511e8b3bf1" containerName="test-curl-7431-140405581687088" Jul 14 19:59:05.898849 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:05.898840 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9cf3a8c-1c6d-46da-895e-00511e8b3bf1" containerName="test-curl-7431-140405581687088" Jul 14 19:59:05.898924 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:05.898918 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9cf3a8c-1c6d-46da-895e-00511e8b3bf1" containerName="test-curl-7431-140405581687088" Jul 14 19:59:05.902152 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:05.902133 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581852720" Jul 14 19:59:05.905147 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:05.905120 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zrwrl\"" Jul 14 19:59:05.907977 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:05.907949 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140405581852720"] Jul 14 19:59:06.027065 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:06.027032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zmzr\" (UniqueName: \"kubernetes.io/projected/bd26bf67-bd18-4ce8-ba55-04b3e8670a84-kube-api-access-6zmzr\") pod \"test-curl-7431-140405581852720\" (UID: \"bd26bf67-bd18-4ce8-ba55-04b3e8670a84\") " pod="opendatahub/test-curl-7431-140405581852720" Jul 14 19:59:06.128054 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:06.128013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zmzr\" (UniqueName: \"kubernetes.io/projected/bd26bf67-bd18-4ce8-ba55-04b3e8670a84-kube-api-access-6zmzr\") pod \"test-curl-7431-140405581852720\" (UID: \"bd26bf67-bd18-4ce8-ba55-04b3e8670a84\") " pod="opendatahub/test-curl-7431-140405581852720" Jul 14 19:59:06.137227 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:06.137195 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zmzr\" (UniqueName: \"kubernetes.io/projected/bd26bf67-bd18-4ce8-ba55-04b3e8670a84-kube-api-access-6zmzr\") pod \"test-curl-7431-140405581852720\" (UID: \"bd26bf67-bd18-4ce8-ba55-04b3e8670a84\") " pod="opendatahub/test-curl-7431-140405581852720" Jul 14 19:59:06.212740 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:06.212640 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581852720" Jul 14 19:59:06.354629 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:06.354576 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7431-140405581852720"] Jul 14 19:59:06.356383 ip-10-0-128-74 kubenswrapper[2576]: W0714 19:59:06.356351 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd26bf67_bd18_4ce8_ba55_04b3e8670a84.slice/crio-d349d7330332b150cfdfc179f30d338bd4a3b79f3d7dd00f0e8397bee52d2e9f WatchSource:0}: Error finding container d349d7330332b150cfdfc179f30d338bd4a3b79f3d7dd00f0e8397bee52d2e9f: Status 404 returned error can't find the container with id d349d7330332b150cfdfc179f30d338bd4a3b79f3d7dd00f0e8397bee52d2e9f Jul 14 19:59:06.463683 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:06.463595 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140405581852720" event={"ID":"bd26bf67-bd18-4ce8-ba55-04b3e8670a84","Type":"ContainerStarted","Data":"d349d7330332b150cfdfc179f30d338bd4a3b79f3d7dd00f0e8397bee52d2e9f"} Jul 14 19:59:07.470259 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:07.470166 2576 generic.go:356] "Generic (PLEG): container finished" podID="bd26bf67-bd18-4ce8-ba55-04b3e8670a84" containerID="4b0a5514bcf265150107013a042d4564df221ebabc237a6b1f6bb9d466ff9843" exitCode=0 Jul 14 19:59:07.470259 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:07.470205 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7431-140405581852720" event={"ID":"bd26bf67-bd18-4ce8-ba55-04b3e8670a84","Type":"ContainerDied","Data":"4b0a5514bcf265150107013a042d4564df221ebabc237a6b1f6bb9d466ff9843"} Jul 14 19:59:08.612719 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:08.612691 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581852720" Jul 14 19:59:08.633347 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:08.633315 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7431-140405581852720_bd26bf67-bd18-4ce8-ba55-04b3e8670a84/test-curl-7431-140405581852720/0.log" Jul 14 19:59:08.658411 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:08.658376 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7431-140405581852720"] Jul 14 19:59:08.660484 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:08.660462 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7431-140405581852720"] Jul 14 19:59:08.753137 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:08.753038 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zmzr\" (UniqueName: \"kubernetes.io/projected/bd26bf67-bd18-4ce8-ba55-04b3e8670a84-kube-api-access-6zmzr\") pod \"bd26bf67-bd18-4ce8-ba55-04b3e8670a84\" (UID: \"bd26bf67-bd18-4ce8-ba55-04b3e8670a84\") " Jul 14 19:59:08.755189 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:08.755164 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd26bf67-bd18-4ce8-ba55-04b3e8670a84-kube-api-access-6zmzr" (OuterVolumeSpecName: "kube-api-access-6zmzr") pod "bd26bf67-bd18-4ce8-ba55-04b3e8670a84" (UID: "bd26bf67-bd18-4ce8-ba55-04b3e8670a84"). InnerVolumeSpecName "kube-api-access-6zmzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:59:08.854000 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:08.853964 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zmzr\" (UniqueName: \"kubernetes.io/projected/bd26bf67-bd18-4ce8-ba55-04b3e8670a84-kube-api-access-6zmzr\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:59:09.482416 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:09.482381 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d349d7330332b150cfdfc179f30d338bd4a3b79f3d7dd00f0e8397bee52d2e9f" Jul 14 19:59:09.482416 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:09.482402 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7431-140405581852720" Jul 14 19:59:09.493374 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:59:09.493339 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7431-140405581852720\" is forbidden: User \"system:node:ip-10-0-128-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-74.ec2.internal' and this object" podUID="bd26bf67-bd18-4ce8-ba55-04b3e8670a84" pod="opendatahub/test-curl-7431-140405581852720" Jul 14 19:59:09.522497 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:09.522457 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd26bf67-bd18-4ce8-ba55-04b3e8670a84" path="/var/lib/kubelet/pods/bd26bf67-bd18-4ce8-ba55-04b3e8670a84/volumes" Jul 14 19:59:43.782463 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:43.782429 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c"] Jul 14 19:59:43.783037 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:43.782721 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" podUID="a8359799-f862-404f-bfb6-a3a018233b9a" containerName="istio-proxy" containerID="cri-o://7e50b1e3f0474535c111062884c98ba0d083c3fe34ac99e57218429ab3bd3b5c" gracePeriod=30 Jul 14 19:59:48.245015 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:48.244979 2576 patch_prober.go:28] interesting pod/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 19:59:48.245475 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:48.245033 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" podUID="a8359799-f862-404f-bfb6-a3a018233b9a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 19:59:49.040663 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.040616 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:59:49.068865 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.068774 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8359799-f862-404f-bfb6-a3a018233b9a-istiod-ca-cert\") pod \"a8359799-f862-404f-bfb6-a3a018233b9a\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " Jul 14 19:59:49.069005 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.068883 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-credential-socket\") pod \"a8359799-f862-404f-bfb6-a3a018233b9a\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " Jul 14 19:59:49.069005 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.068919 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-workload-certs\") pod \"a8359799-f862-404f-bfb6-a3a018233b9a\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " Jul 14 19:59:49.069005 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.068964 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-workload-socket\") pod \"a8359799-f862-404f-bfb6-a3a018233b9a\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " Jul 14 19:59:49.069172 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.069012 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zs8h\" (UniqueName: \"kubernetes.io/projected/a8359799-f862-404f-bfb6-a3a018233b9a-kube-api-access-2zs8h\") pod \"a8359799-f862-404f-bfb6-a3a018233b9a\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " Jul 14 19:59:49.069172 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.069057 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-istio-data\") pod \"a8359799-f862-404f-bfb6-a3a018233b9a\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " Jul 14 19:59:49.069172 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.069082 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8359799-f862-404f-bfb6-a3a018233b9a-istio-token\") pod \"a8359799-f862-404f-bfb6-a3a018233b9a\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " Jul 14 19:59:49.069172 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.069110 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8359799-f862-404f-bfb6-a3a018233b9a-istio-podinfo\") pod \"a8359799-f862-404f-bfb6-a3a018233b9a\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " Jul 14 19:59:49.069172 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.069134 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a8359799-f862-404f-bfb6-a3a018233b9a" (UID: "a8359799-f862-404f-bfb6-a3a018233b9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:59:49.069172 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.069147 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-istio-envoy\") pod \"a8359799-f862-404f-bfb6-a3a018233b9a\" (UID: \"a8359799-f862-404f-bfb6-a3a018233b9a\") " Jul 14 19:59:49.069474 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.069192 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8359799-f862-404f-bfb6-a3a018233b9a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a8359799-f862-404f-bfb6-a3a018233b9a" (UID: "a8359799-f862-404f-bfb6-a3a018233b9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:59:49.069474 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.069282 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a8359799-f862-404f-bfb6-a3a018233b9a" (UID: "a8359799-f862-404f-bfb6-a3a018233b9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:59:49.069474 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.069296 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a8359799-f862-404f-bfb6-a3a018233b9a" (UID: "a8359799-f862-404f-bfb6-a3a018233b9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:59:49.069690 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.069558 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8359799-f862-404f-bfb6-a3a018233b9a-istiod-ca-cert\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:59:49.069690 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.069576 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-credential-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:59:49.069690 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.069608 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-workload-certs\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:59:49.069690 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.069621 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-workload-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:59:49.070258 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.070233 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-istio-data" (OuterVolumeSpecName: "istio-data") pod "a8359799-f862-404f-bfb6-a3a018233b9a" (UID: "a8359799-f862-404f-bfb6-a3a018233b9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:59:49.071320 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.071291 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8359799-f862-404f-bfb6-a3a018233b9a-kube-api-access-2zs8h" (OuterVolumeSpecName: "kube-api-access-2zs8h") pod "a8359799-f862-404f-bfb6-a3a018233b9a" (UID: "a8359799-f862-404f-bfb6-a3a018233b9a"). InnerVolumeSpecName "kube-api-access-2zs8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:59:49.071417 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.071402 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8359799-f862-404f-bfb6-a3a018233b9a-istio-token" (OuterVolumeSpecName: "istio-token") pod "a8359799-f862-404f-bfb6-a3a018233b9a" (UID: "a8359799-f862-404f-bfb6-a3a018233b9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:59:49.071495 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.071475 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8359799-f862-404f-bfb6-a3a018233b9a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a8359799-f862-404f-bfb6-a3a018233b9a" (UID: "a8359799-f862-404f-bfb6-a3a018233b9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 19:59:49.071791 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.071769 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a8359799-f862-404f-bfb6-a3a018233b9a" (UID: "a8359799-f862-404f-bfb6-a3a018233b9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:59:49.170471 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.170436 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-istio-envoy\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:59:49.170471 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.170466 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zs8h\" (UniqueName: \"kubernetes.io/projected/a8359799-f862-404f-bfb6-a3a018233b9a-kube-api-access-2zs8h\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:59:49.170471 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.170475 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8359799-f862-404f-bfb6-a3a018233b9a-istio-data\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:59:49.170471 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.170483 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8359799-f862-404f-bfb6-a3a018233b9a-istio-token\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:59:49.170767 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.170491 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8359799-f862-404f-bfb6-a3a018233b9a-istio-podinfo\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 19:59:49.696160 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.696125 2576 generic.go:356] "Generic (PLEG): container finished" podID="a8359799-f862-404f-bfb6-a3a018233b9a" containerID="7e50b1e3f0474535c111062884c98ba0d083c3fe34ac99e57218429ab3bd3b5c" exitCode=0 Jul 14 19:59:49.696645 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.696173 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" Jul 14 19:59:49.696645 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.696216 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" event={"ID":"a8359799-f862-404f-bfb6-a3a018233b9a","Type":"ContainerDied","Data":"7e50b1e3f0474535c111062884c98ba0d083c3fe34ac99e57218429ab3bd3b5c"} Jul 14 19:59:49.696645 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.696271 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c" event={"ID":"a8359799-f862-404f-bfb6-a3a018233b9a","Type":"ContainerDied","Data":"50e67c6168abb7fac51ff979f7995ae7f7a868c7edb40d30d3190dc19a8d8fbf"} Jul 14 19:59:49.696645 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.696296 2576 scope.go:117] "RemoveContainer" containerID="7e50b1e3f0474535c111062884c98ba0d083c3fe34ac99e57218429ab3bd3b5c" Jul 14 19:59:49.705445 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.705425 2576 scope.go:117] "RemoveContainer" containerID="5be89fb78d3527ee4b4b49558f00c7b8603f0da2aa65a509ead4fca0647a588b" Jul 14 19:59:49.714744 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.714723 2576 scope.go:117] "RemoveContainer" containerID="7e50b1e3f0474535c111062884c98ba0d083c3fe34ac99e57218429ab3bd3b5c" Jul 14 19:59:49.715087 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:59:49.715054 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e50b1e3f0474535c111062884c98ba0d083c3fe34ac99e57218429ab3bd3b5c\": container with ID starting with 7e50b1e3f0474535c111062884c98ba0d083c3fe34ac99e57218429ab3bd3b5c not found: ID does not exist" containerID="7e50b1e3f0474535c111062884c98ba0d083c3fe34ac99e57218429ab3bd3b5c" Jul 14 19:59:49.715147 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.715104 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e50b1e3f0474535c111062884c98ba0d083c3fe34ac99e57218429ab3bd3b5c"} err="failed to get container status \"7e50b1e3f0474535c111062884c98ba0d083c3fe34ac99e57218429ab3bd3b5c\": rpc error: code = NotFound desc = could not find container \"7e50b1e3f0474535c111062884c98ba0d083c3fe34ac99e57218429ab3bd3b5c\": container with ID starting with 7e50b1e3f0474535c111062884c98ba0d083c3fe34ac99e57218429ab3bd3b5c not found: ID does not exist" Jul 14 19:59:49.715147 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.715130 2576 scope.go:117] "RemoveContainer" containerID="5be89fb78d3527ee4b4b49558f00c7b8603f0da2aa65a509ead4fca0647a588b" Jul 14 19:59:49.715409 ip-10-0-128-74 kubenswrapper[2576]: E0714 19:59:49.715389 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5be89fb78d3527ee4b4b49558f00c7b8603f0da2aa65a509ead4fca0647a588b\": container with ID starting with 5be89fb78d3527ee4b4b49558f00c7b8603f0da2aa65a509ead4fca0647a588b not found: ID does not exist" containerID="5be89fb78d3527ee4b4b49558f00c7b8603f0da2aa65a509ead4fca0647a588b" Jul 14 19:59:49.715470 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.715419 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be89fb78d3527ee4b4b49558f00c7b8603f0da2aa65a509ead4fca0647a588b"} err="failed to get container status \"5be89fb78d3527ee4b4b49558f00c7b8603f0da2aa65a509ead4fca0647a588b\": rpc error: code = NotFound desc = could not find container \"5be89fb78d3527ee4b4b49558f00c7b8603f0da2aa65a509ead4fca0647a588b\": container with ID starting with 5be89fb78d3527ee4b4b49558f00c7b8603f0da2aa65a509ead4fca0647a588b not found: ID does not exist" Jul 14 19:59:49.716043 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.716016 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c"] Jul 14 19:59:49.718614 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:49.718596 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-70659c-openshift-default-57dfc9f647-d2j9c"] Jul 14 19:59:51.515674 ip-10-0-128-74 kubenswrapper[2576]: I0714 19:59:51.515644 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8359799-f862-404f-bfb6-a3a018233b9a" path="/var/lib/kubelet/pods/a8359799-f862-404f-bfb6-a3a018233b9a/volumes" Jul 14 20:00:09.983664 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:09.983564 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f"] Jul 14 20:00:09.984399 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:09.983815 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" podUID="092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" containerName="istio-proxy" containerID="cri-o://bd23f17e22f1c9c0c2eed5bc1e24f54184f2b84148183e3978739e2947a099e9" gracePeriod=30 Jul 14 20:00:15.243107 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.243069 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 20:00:15.324463 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.324436 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-podinfo\") pod \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " Jul 14 20:00:15.324662 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.324473 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-credential-socket\") pod \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " Jul 14 20:00:15.324662 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.324508 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-token\") pod \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " Jul 14 20:00:15.324662 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.324531 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istiod-ca-cert\") pod \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " Jul 14 20:00:15.324662 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.324557 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-envoy\") pod \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " Jul 14 20:00:15.324662 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.324604 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxfht\" (UniqueName: \"kubernetes.io/projected/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-kube-api-access-wxfht\") pod \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " Jul 14 20:00:15.324662 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.324630 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-data\") pod \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " Jul 14 20:00:15.324985 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.324686 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-workload-certs\") pod \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " Jul 14 20:00:15.324985 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.324740 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-workload-socket\") pod \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\" (UID: \"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b\") " Jul 14 20:00:15.324985 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.324866 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" (UID: "092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:00:15.325151 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.325045 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" (UID: "092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:00:15.325151 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.325089 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" (UID: "092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:00:15.325263 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.325177 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-workload-certs\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 20:00:15.325263 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.325194 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-credential-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 20:00:15.325263 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.325216 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" (UID: "092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:00:15.325548 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.325524 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-data" (OuterVolumeSpecName: "istio-data") pod "092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" (UID: "092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:00:15.326844 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.326824 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" (UID: "092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:00:15.326977 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.326963 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-token" (OuterVolumeSpecName: "istio-token") pod "092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" (UID: "092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:00:15.327167 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.327144 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-kube-api-access-wxfht" (OuterVolumeSpecName: "kube-api-access-wxfht") pod "092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" (UID: "092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b"). InnerVolumeSpecName "kube-api-access-wxfht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:00:15.327234 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.327154 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" (UID: "092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:00:15.426009 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.425971 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-podinfo\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 20:00:15.426009 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.426004 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-token\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 20:00:15.426009 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.426014 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istiod-ca-cert\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 20:00:15.426009 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.426022 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-envoy\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 20:00:15.426283 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.426030 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxfht\" (UniqueName: \"kubernetes.io/projected/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-kube-api-access-wxfht\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 20:00:15.426283 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.426040 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-istio-data\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 20:00:15.426283 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.426049 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b-workload-socket\") on node \"ip-10-0-128-74.ec2.internal\" DevicePath \"\"" Jul 14 20:00:15.838569 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.838537 2576 generic.go:356] "Generic (PLEG): container finished" podID="092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" containerID="bd23f17e22f1c9c0c2eed5bc1e24f54184f2b84148183e3978739e2947a099e9" exitCode=0 Jul 14 20:00:15.838755 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.838619 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" Jul 14 20:00:15.838755 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.838617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" event={"ID":"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b","Type":"ContainerDied","Data":"bd23f17e22f1c9c0c2eed5bc1e24f54184f2b84148183e3978739e2947a099e9"} Jul 14 20:00:15.838755 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.838705 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f" event={"ID":"092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b","Type":"ContainerDied","Data":"b355b7860c2250fbf2bc300181e335053d26508f8b729c5906dfaa29ab136681"} Jul 14 20:00:15.838755 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.838729 2576 scope.go:117] "RemoveContainer" containerID="bd23f17e22f1c9c0c2eed5bc1e24f54184f2b84148183e3978739e2947a099e9" Jul 14 20:00:15.847526 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.847510 2576 scope.go:117] "RemoveContainer" containerID="294a0a5baf585f0971554a3bf763e33e8e6d57cca2b67a3f4c78f1884a48fe5c" Jul 14 20:00:15.855849 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.855815 2576 scope.go:117] "RemoveContainer" containerID="bd23f17e22f1c9c0c2eed5bc1e24f54184f2b84148183e3978739e2947a099e9" Jul 14 20:00:15.856107 ip-10-0-128-74 kubenswrapper[2576]: E0714 20:00:15.856084 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd23f17e22f1c9c0c2eed5bc1e24f54184f2b84148183e3978739e2947a099e9\": container with ID starting with bd23f17e22f1c9c0c2eed5bc1e24f54184f2b84148183e3978739e2947a099e9 not found: ID does not exist" containerID="bd23f17e22f1c9c0c2eed5bc1e24f54184f2b84148183e3978739e2947a099e9" Jul 14 20:00:15.856107 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.856115 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd23f17e22f1c9c0c2eed5bc1e24f54184f2b84148183e3978739e2947a099e9"} err="failed to get container status \"bd23f17e22f1c9c0c2eed5bc1e24f54184f2b84148183e3978739e2947a099e9\": rpc error: code = NotFound desc = could not find container \"bd23f17e22f1c9c0c2eed5bc1e24f54184f2b84148183e3978739e2947a099e9\": container with ID starting with bd23f17e22f1c9c0c2eed5bc1e24f54184f2b84148183e3978739e2947a099e9 not found: ID does not exist" Jul 14 20:00:15.856107 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.856133 2576 scope.go:117] "RemoveContainer" containerID="294a0a5baf585f0971554a3bf763e33e8e6d57cca2b67a3f4c78f1884a48fe5c" Jul 14 20:00:15.856355 ip-10-0-128-74 kubenswrapper[2576]: E0714 20:00:15.856337 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"294a0a5baf585f0971554a3bf763e33e8e6d57cca2b67a3f4c78f1884a48fe5c\": container with ID starting with 294a0a5baf585f0971554a3bf763e33e8e6d57cca2b67a3f4c78f1884a48fe5c not found: ID does not exist" containerID="294a0a5baf585f0971554a3bf763e33e8e6d57cca2b67a3f4c78f1884a48fe5c" Jul 14 20:00:15.856399 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.856359 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"294a0a5baf585f0971554a3bf763e33e8e6d57cca2b67a3f4c78f1884a48fe5c"} err="failed to get container status \"294a0a5baf585f0971554a3bf763e33e8e6d57cca2b67a3f4c78f1884a48fe5c\": rpc error: code = NotFound desc = could not find container \"294a0a5baf585f0971554a3bf763e33e8e6d57cca2b67a3f4c78f1884a48fe5c\": container with ID starting with 294a0a5baf585f0971554a3bf763e33e8e6d57cca2b67a3f4c78f1884a48fe5c not found: ID does not exist" Jul 14 20:00:15.857960 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.857941 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f"] Jul 14 20:00:15.862319 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:15.862298 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-356b53-openshift-default-6f9f9bb4ff-bkn5f"] Jul 14 20:00:16.792461 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:16.792427 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f9786d97d-vjr9z_2506a3e9-589c-479e-8b3e-37d879abbf36/manager/0.log" Jul 14 20:00:16.800776 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:16.800755 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f9786d97d-vjr9z_2506a3e9-589c-479e-8b3e-37d879abbf36/copy-manifests/0.log" Jul 14 20:00:17.515568 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:17.515534 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" path="/var/lib/kubelet/pods/092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b/volumes" Jul 14 20:00:17.675809 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:17.675780 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-102a2069-zs75k_8afc9462-76c0-42cc-9f56-edd8a76bec85/revoke-keys/0.log" Jul 14 20:00:17.795752 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:17.795672 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-f61c44f4-sk658_9c15e78b-6940-4143-a1b2-e3cf4ae30c3f/revoke-keys/1.log" Jul 14 20:00:17.920322 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:17.920291 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-922b98-xr9gk_46009e58-41df-4454-be6f-159f1affa9e5/revoke-keys/0.log" Jul 14 20:00:18.045112 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:18.045083 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-57e988-g6fzr_d926bbd3-7f24-40c2-893f-1e48d9d9bbd9/revoke-keys/0.log" Jul 14 20:00:18.165931 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:18.165904 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-9d072882-cgltx_1a0025f4-d92d-4a08-b9ae-18686a8ae2b6/revoke-keys/0.log" Jul 14 20:00:18.527111 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:18.527016 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-0bc32ee5-45dsc_54335b12-dfff-4cc9-9378-352099972dbc/revoke-keys/1.log" Jul 14 20:00:18.645109 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:18.645081 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-9b84c5bc-hkqpt_3787b1bc-7c59-42b7-8f60-8b1dac53a2b7/revoke-keys/0.log" Jul 14 20:00:18.764954 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:18.764928 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-ea441188-qwb2x_4f83598f-2c57-46b1-abe7-87aef32d89a6/revoke-keys/0.log" Jul 14 20:00:19.122156 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:19.122128 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-nfzrx_33693bfd-f953-4e4d-b668-b478d36ad79c/postgres/0.log" Jul 14 20:00:20.761857 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:20.761833 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-m7btf_5a960dcd-05a0-4e66-af82-ee4f975182e2/manager/0.log" Jul 14 20:00:20.879435 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:20.879404 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-xjwwz_bc50cefe-792d-4e62-bb65-bb493dce3b7f/kuadrant-console-plugin/0.log" Jul 14 20:00:21.003154 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:21.003120 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-rhsms_80cda187-571e-49c8-b132-d735ac735b5f/registry-server/0.log" Jul 14 20:00:21.133137 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:21.133101 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-kxn8z_1c13f2e4-d0b8-4ca5-8f34-2238c7060164/manager/0.log" Jul 14 20:00:21.381813 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:21.381766 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-m55b7_a36da07c-6fdc-4ee1-afa9-27e6ac38396b/manager/0.log" Jul 14 20:00:22.284698 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:22.284666 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-fcpcm_3e3ab25f-fc0d-4d53-97c7-48048ff7a532/istio-proxy/1.log" Jul 14 20:00:22.665125 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:22.665093 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79c9798784-bdl9b_01ac54b9-9511-4e3c-afcd-482292702a0b/router/0.log" Jul 14 20:00:23.165497 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:23.165466 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-vt94l_b3593aa0-22b9-42de-a13f-1ca28558e244/main/0.log" Jul 14 20:00:23.408812 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:23.408782 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpbv7s_5694bb63-fad1-466f-8295-c041cef8b0c3/main/0.log" Jul 14 20:00:30.939731 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:30.939699 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bv2wz_5e6c415f-305f-477e-b939-1c2b84a3d770/global-pull-secret-syncer/0.log" Jul 14 20:00:31.099395 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:31.099365 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bb6wd_659ae1d5-f1bd-4bec-b2c9-4b5fe3ae0a8d/konnectivity-agent/0.log" Jul 14 20:00:31.152065 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:31.152035 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-74.ec2.internal_cc14085ea90483c50f2a94112a007054/haproxy/0.log" Jul 14 20:00:36.135413 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:36.135376 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-m7btf_5a960dcd-05a0-4e66-af82-ee4f975182e2/manager/0.log" Jul 14 20:00:36.159853 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:36.159826 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-xjwwz_bc50cefe-792d-4e62-bb65-bb493dce3b7f/kuadrant-console-plugin/0.log" Jul 14 20:00:36.197865 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:36.197831 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-rhsms_80cda187-571e-49c8-b132-d735ac735b5f/registry-server/0.log" Jul 14 20:00:36.303527 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:36.303490 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-kxn8z_1c13f2e4-d0b8-4ca5-8f34-2238c7060164/manager/0.log" Jul 14 20:00:36.386892 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:36.386797 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-m55b7_a36da07c-6fdc-4ee1-afa9-27e6ac38396b/manager/0.log" Jul 14 20:00:38.096212 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:38.096181 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-4mvrd_c90a7de7-c072-40bd-bebc-ee4996a2e763/cluster-monitoring-operator/0.log" Jul 14 20:00:38.141615 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:38.141576 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-hrqw7_13abf531-dedb-495b-9d91-96c55586b544/kube-state-metrics/0.log" Jul 14 20:00:38.180279 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:38.180250 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-hrqw7_13abf531-dedb-495b-9d91-96c55586b544/kube-rbac-proxy-main/0.log" Jul 14 20:00:38.207063 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:38.207039 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-hrqw7_13abf531-dedb-495b-9d91-96c55586b544/kube-rbac-proxy-self/0.log" Jul 14 20:00:38.480766 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:38.480735 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zz95x_fd24d679-7f30-4b07-9d49-24f9f81bfb58/node-exporter/0.log" Jul 14 20:00:38.503153 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:38.503124 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zz95x_fd24d679-7f30-4b07-9d49-24f9f81bfb58/kube-rbac-proxy/0.log" Jul 14 20:00:38.526793 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:38.526773 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zz95x_fd24d679-7f30-4b07-9d49-24f9f81bfb58/init-textfile/0.log" Jul 14 20:00:38.560501 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:38.560473 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-zpfsp_db281d62-b537-4268-b9f5-7946d58f1daf/kube-rbac-proxy-main/0.log" Jul 14 20:00:38.583541 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:38.583511 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-zpfsp_db281d62-b537-4268-b9f5-7946d58f1daf/kube-rbac-proxy-self/0.log" Jul 14 20:00:38.604420 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:38.604393 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-zpfsp_db281d62-b537-4268-b9f5-7946d58f1daf/openshift-state-metrics/0.log" Jul 14 20:00:38.640976 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:38.640945 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8369730-632e-42e9-9083-fc0027edf7fc/prometheus/0.log" Jul 14 20:00:38.671977 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:38.671953 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8369730-632e-42e9-9083-fc0027edf7fc/config-reloader/0.log" Jul 14 20:00:38.694307 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:38.694281 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8369730-632e-42e9-9083-fc0027edf7fc/thanos-sidecar/0.log" Jul 14 20:00:38.714446 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:38.714423 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8369730-632e-42e9-9083-fc0027edf7fc/kube-rbac-proxy-web/0.log" Jul 14 20:00:38.738492 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:38.738414 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8369730-632e-42e9-9083-fc0027edf7fc/kube-rbac-proxy/0.log" Jul 14 20:00:38.760305 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:38.760279 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8369730-632e-42e9-9083-fc0027edf7fc/kube-rbac-proxy-thanos/0.log" Jul 14 20:00:38.787863 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:38.787840 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8369730-632e-42e9-9083-fc0027edf7fc/init-config-reloader/0.log" Jul 14 20:00:41.560557 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:41.560522 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-c59d87654-pms5h_e58163a1-1cfb-469f-8fec-0f421faa3932/console/0.log" Jul 14 20:00:43.159006 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:43.158971 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-29wvt_f3ea0e40-5a86-4c4b-980b-2007a1290795/dns/0.log" Jul 14 20:00:43.177874 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:43.177846 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-29wvt_f3ea0e40-5a86-4c4b-980b-2007a1290795/kube-rbac-proxy/0.log" Jul 14 20:00:43.306876 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:43.306839 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q6rws_e0a0b990-bc45-4764-bd78-31069184c58a/dns-node-resolver/0.log" Jul 14 20:00:43.843487 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:43.843453 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5b4f8cdf57-hdqch_44fffc73-9677-466d-a63c-7ba76f31f8f5/registry/0.log" Jul 14 20:00:43.907418 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:43.907384 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z4lnk_70375e8e-ae6d-4ce1-8240-012361054757/node-ca/0.log" Jul 14 20:00:44.393512 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.393471 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r"] Jul 14 20:00:44.393915 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.393875 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd26bf67-bd18-4ce8-ba55-04b3e8670a84" containerName="test-curl-7431-140405581852720" Jul 14 20:00:44.393915 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.393889 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd26bf67-bd18-4ce8-ba55-04b3e8670a84" containerName="test-curl-7431-140405581852720" Jul 14 20:00:44.393915 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.393912 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8359799-f862-404f-bfb6-a3a018233b9a" containerName="istio-proxy" Jul 14 20:00:44.394088 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.393918 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8359799-f862-404f-bfb6-a3a018233b9a" containerName="istio-proxy" Jul 14 20:00:44.394088 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.393931 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" containerName="istio-proxy" Jul 14 20:00:44.394088 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.393940 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" containerName="istio-proxy" Jul 14 20:00:44.394088 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.393948 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8359799-f862-404f-bfb6-a3a018233b9a" containerName="istio-proxy" Jul 14 20:00:44.394088 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.393954 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8359799-f862-404f-bfb6-a3a018233b9a" containerName="istio-proxy" Jul 14 20:00:44.394088 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.393961 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" containerName="istio-proxy" Jul 14 20:00:44.394088 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.393966 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" containerName="istio-proxy" Jul 14 20:00:44.394088 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.394022 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" containerName="istio-proxy" Jul 14 20:00:44.394088 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.394031 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="092f7b9b-bc8d-426c-b828-1c8cdbb9ea1b" containerName="istio-proxy" Jul 14 20:00:44.394088 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.394040 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8359799-f862-404f-bfb6-a3a018233b9a" containerName="istio-proxy" Jul 14 20:00:44.394088 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.394046 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd26bf67-bd18-4ce8-ba55-04b3e8670a84" containerName="test-curl-7431-140405581852720" Jul 14 20:00:44.397448 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.397432 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:44.400507 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.400485 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bmp8z\"/\"kube-root-ca.crt\"" Jul 14 20:00:44.400639 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.400550 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bmp8z\"/\"default-dockercfg-skmfs\"" Jul 14 20:00:44.400696 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.400639 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bmp8z\"/\"openshift-service-ca.crt\"" Jul 14 20:00:44.409643 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.409611 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r"] Jul 14 20:00:44.505222 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.505189 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2f707e8-ed22-4861-ac40-7e8e39e74455-sys\") pod \"perf-node-gather-daemonset-cft8r\" (UID: \"d2f707e8-ed22-4861-ac40-7e8e39e74455\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:44.505222 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.505229 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d2f707e8-ed22-4861-ac40-7e8e39e74455-podres\") pod \"perf-node-gather-daemonset-cft8r\" (UID: \"d2f707e8-ed22-4861-ac40-7e8e39e74455\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:44.505482 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.505270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2f707e8-ed22-4861-ac40-7e8e39e74455-lib-modules\") pod \"perf-node-gather-daemonset-cft8r\" (UID: \"d2f707e8-ed22-4861-ac40-7e8e39e74455\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:44.505482 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.505294 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d2f707e8-ed22-4861-ac40-7e8e39e74455-proc\") pod \"perf-node-gather-daemonset-cft8r\" (UID: \"d2f707e8-ed22-4861-ac40-7e8e39e74455\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:44.505482 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.505468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w7rm\" (UniqueName: \"kubernetes.io/projected/d2f707e8-ed22-4861-ac40-7e8e39e74455-kube-api-access-8w7rm\") pod \"perf-node-gather-daemonset-cft8r\" (UID: \"d2f707e8-ed22-4861-ac40-7e8e39e74455\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:44.606533 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.606501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w7rm\" (UniqueName: \"kubernetes.io/projected/d2f707e8-ed22-4861-ac40-7e8e39e74455-kube-api-access-8w7rm\") pod \"perf-node-gather-daemonset-cft8r\" (UID: \"d2f707e8-ed22-4861-ac40-7e8e39e74455\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:44.606533 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.606539 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2f707e8-ed22-4861-ac40-7e8e39e74455-sys\") pod \"perf-node-gather-daemonset-cft8r\" (UID: \"d2f707e8-ed22-4861-ac40-7e8e39e74455\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:44.606768 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.606556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d2f707e8-ed22-4861-ac40-7e8e39e74455-podres\") pod \"perf-node-gather-daemonset-cft8r\" (UID: \"d2f707e8-ed22-4861-ac40-7e8e39e74455\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:44.606768 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.606606 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2f707e8-ed22-4861-ac40-7e8e39e74455-lib-modules\") pod \"perf-node-gather-daemonset-cft8r\" (UID: \"d2f707e8-ed22-4861-ac40-7e8e39e74455\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:44.606768 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.606634 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d2f707e8-ed22-4861-ac40-7e8e39e74455-proc\") pod \"perf-node-gather-daemonset-cft8r\" (UID: \"d2f707e8-ed22-4861-ac40-7e8e39e74455\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:44.606768 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.606652 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2f707e8-ed22-4861-ac40-7e8e39e74455-sys\") pod \"perf-node-gather-daemonset-cft8r\" (UID: \"d2f707e8-ed22-4861-ac40-7e8e39e74455\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:44.606768 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.606725 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d2f707e8-ed22-4861-ac40-7e8e39e74455-proc\") pod \"perf-node-gather-daemonset-cft8r\" (UID: \"d2f707e8-ed22-4861-ac40-7e8e39e74455\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:44.606768 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.606751 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d2f707e8-ed22-4861-ac40-7e8e39e74455-podres\") pod \"perf-node-gather-daemonset-cft8r\" (UID: \"d2f707e8-ed22-4861-ac40-7e8e39e74455\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:44.606768 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.606754 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2f707e8-ed22-4861-ac40-7e8e39e74455-lib-modules\") pod \"perf-node-gather-daemonset-cft8r\" (UID: \"d2f707e8-ed22-4861-ac40-7e8e39e74455\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:44.614863 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.614837 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w7rm\" (UniqueName: \"kubernetes.io/projected/d2f707e8-ed22-4861-ac40-7e8e39e74455-kube-api-access-8w7rm\") pod \"perf-node-gather-daemonset-cft8r\" (UID: \"d2f707e8-ed22-4861-ac40-7e8e39e74455\") " pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:44.709632 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.709508 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:44.840000 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.839974 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r"] Jul 14 20:00:44.841001 ip-10-0-128-74 kubenswrapper[2576]: W0714 20:00:44.840973 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd2f707e8_ed22_4861_ac40_7e8e39e74455.slice/crio-1bdc7e447c453e6e1068e2e2e249a6e8a2c4e5ee93f6ddb3bd44d467e33626ff WatchSource:0}: Error finding container 1bdc7e447c453e6e1068e2e2e249a6e8a2c4e5ee93f6ddb3bd44d467e33626ff: Status 404 returned error can't find the container with id 1bdc7e447c453e6e1068e2e2e249a6e8a2c4e5ee93f6ddb3bd44d467e33626ff Jul 14 20:00:44.996723 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.996691 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" event={"ID":"d2f707e8-ed22-4861-ac40-7e8e39e74455","Type":"ContainerStarted","Data":"9313b647a827a61da6b40e7a64de640bf7884c1db3656ee088eae352ef6a94a8"} Jul 14 20:00:44.996839 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.996751 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" event={"ID":"d2f707e8-ed22-4861-ac40-7e8e39e74455","Type":"ContainerStarted","Data":"1bdc7e447c453e6e1068e2e2e249a6e8a2c4e5ee93f6ddb3bd44d467e33626ff"} Jul 14 20:00:44.996839 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:44.996781 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:45.014409 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:45.014370 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" podStartSLOduration=1.014356014 podStartE2EDuration="1.014356014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 20:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:00:45.012472015 +0000 UTC m=+2998.125721890" watchObservedRunningTime="2026-07-14 20:00:45.014356014 +0000 UTC m=+2998.127605905" Jul 14 20:00:45.124842 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:45.124815 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-fcpcm_3e3ab25f-fc0d-4d53-97c7-48048ff7a532/istio-proxy/0.log" Jul 14 20:00:45.154150 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:45.154126 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-fcpcm_3e3ab25f-fc0d-4d53-97c7-48048ff7a532/istio-proxy/1.log" Jul 14 20:00:45.238007 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:45.237934 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79c9798784-bdl9b_01ac54b9-9511-4e3c-afcd-482292702a0b/router/0.log" Jul 14 20:00:45.961572 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:45.961542 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j7zhz_5e789e3e-9025-425e-bdfb-b8caedfa2767/serve-healthcheck-canary/0.log" Jul 14 20:00:46.437670 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:46.437641 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-pxckb_e92e640a-57c7-4a12-89cb-701c6f9bf3e6/insights-operator/0.log" Jul 14 20:00:46.438305 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:46.438292 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-pxckb_e92e640a-57c7-4a12-89cb-701c6f9bf3e6/insights-operator/1.log" Jul 14 20:00:46.583614 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:46.583569 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m2qzw_db924ccb-4ed7-4682-98cf-2e696b0d9405/kube-rbac-proxy/0.log" Jul 14 20:00:46.603448 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:46.603421 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m2qzw_db924ccb-4ed7-4682-98cf-2e696b0d9405/exporter/0.log" Jul 14 20:00:46.623639 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:46.623610 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m2qzw_db924ccb-4ed7-4682-98cf-2e696b0d9405/extractor/0.log" Jul 14 20:00:47.662620 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:47.662594 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 20:00:47.681321 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:47.681290 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 20:00:49.221018 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:49.220982 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f9786d97d-vjr9z_2506a3e9-589c-479e-8b3e-37d879abbf36/manager/0.log" Jul 14 20:00:49.237743 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:49.237713 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f9786d97d-vjr9z_2506a3e9-589c-479e-8b3e-37d879abbf36/copy-manifests/0.log" Jul 14 20:00:50.445362 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:50.445333 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-59cf65bbd6-wnvgs_c2cbdf74-1983-4fa3-b716-04cbbdf805b4/manager/0.log" Jul 14 20:00:51.013171 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:51.013145 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bmp8z/perf-node-gather-daemonset-cft8r" Jul 14 20:00:51.507308 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:51.507271 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-fcpcm_3e3ab25f-fc0d-4d53-97c7-48048ff7a532/istio-proxy/0.log" Jul 14 20:00:51.537785 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:51.537754 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-fcpcm_3e3ab25f-fc0d-4d53-97c7-48048ff7a532/istio-proxy/1.log" Jul 14 20:00:51.604026 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:51.603998 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79c9798784-bdl9b_01ac54b9-9511-4e3c-afcd-482292702a0b/router/0.log" Jul 14 20:00:52.875220 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:52.875181 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-fcpcm_3e3ab25f-fc0d-4d53-97c7-48048ff7a532/istio-proxy/0.log" Jul 14 20:00:52.903319 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:52.903292 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-fcpcm_3e3ab25f-fc0d-4d53-97c7-48048ff7a532/istio-proxy/1.log" Jul 14 20:00:52.972347 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:52.972320 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79c9798784-bdl9b_01ac54b9-9511-4e3c-afcd-482292702a0b/router/0.log" Jul 14 20:00:56.144637 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:56.144604 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-dnr7b_b20459ce-1f34-4dcd-becb-522fee08d273/kube-storage-version-migrator-operator/1.log" Jul 14 20:00:56.145274 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:56.145246 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-dnr7b_b20459ce-1f34-4dcd-becb-522fee08d273/kube-storage-version-migrator-operator/0.log" Jul 14 20:00:57.162452 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:57.162425 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f7zqk_2d475a4f-8acf-4cc1-b605-be9c1be74d34/kube-multus-additional-cni-plugins/0.log" Jul 14 20:00:57.183387 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:57.183358 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f7zqk_2d475a4f-8acf-4cc1-b605-be9c1be74d34/egress-router-binary-copy/0.log" Jul 14 20:00:57.202950 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:57.202919 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f7zqk_2d475a4f-8acf-4cc1-b605-be9c1be74d34/cni-plugins/0.log" Jul 14 20:00:57.224505 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:57.224472 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f7zqk_2d475a4f-8acf-4cc1-b605-be9c1be74d34/bond-cni-plugin/0.log" Jul 14 20:00:57.245338 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:57.245306 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f7zqk_2d475a4f-8acf-4cc1-b605-be9c1be74d34/routeoverride-cni/0.log" Jul 14 20:00:57.270358 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:57.270329 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f7zqk_2d475a4f-8acf-4cc1-b605-be9c1be74d34/whereabouts-cni-bincopy/0.log" Jul 14 20:00:57.293104 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:57.293075 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f7zqk_2d475a4f-8acf-4cc1-b605-be9c1be74d34/whereabouts-cni/0.log" Jul 14 20:00:57.640649 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:57.640623 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pxrgt_e563268e-31ea-4ddb-82ff-da5017d5a3fb/kube-multus/0.log" Jul 14 20:00:57.776971 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:57.776922 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xczln_8d1706ff-a844-42e9-b6d8-628495d88055/network-metrics-daemon/0.log" Jul 14 20:00:57.792065 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:57.792042 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xczln_8d1706ff-a844-42e9-b6d8-628495d88055/kube-rbac-proxy/0.log" Jul 14 20:00:58.789732 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:58.789705 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-controller/0.log" Jul 14 20:00:58.804896 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:58.804868 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/1.log" Jul 14 20:00:58.804896 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:58.804889 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovn-acl-logging/0.log" Jul 14 20:00:58.825607 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:58.825559 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/kube-rbac-proxy-node/0.log" Jul 14 20:00:58.845651 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:58.845614 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/kube-rbac-proxy-ovn-metrics/0.log" Jul 14 20:00:58.863128 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:58.863098 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/northd/0.log" Jul 14 20:00:58.881835 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:58.881804 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/nbdb/0.log" Jul 14 20:00:58.900826 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:58.900794 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/sbdb/0.log" Jul 14 20:00:58.957772 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:00:58.957741 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg2x9_dd9dbd61-c491-4f7a-81b7-1769a9e2cb02/ovnkube-controller/0.log" Jul 14 20:01:00.204003 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:01:00.203976 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-869dcfcdcd-tt2st_63ff1a4c-4415-486c-a331-e5a37b4cbaaa/check-endpoints/0.log" Jul 14 20:01:00.248822 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:01:00.248795 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-tgldx_67bb23c7-9bee-4350-a4bf-0da1589fbce0/network-check-target-container/0.log" Jul 14 20:01:01.374335 ip-10-0-128-74 kubenswrapper[2576]: I0714 20:01:01.374304 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-msbnp_ec17ab27-ef49-4fae-9006-b236a56a1727/iptables-alerter/0.log"