Jul 21 21:48:42.123666 ip-10-0-135-158 systemd[1]: Starting Kubernetes Kubelet... Jul 21 21:48:42.501728 ip-10-0-135-158 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 21:48:42.501728 ip-10-0-135-158 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 21:48:42.501728 ip-10-0-135-158 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 21:48:42.501728 ip-10-0-135-158 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 21:48:42.501728 ip-10-0-135-158 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 21:48:42.503519 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.503403 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 21:48:42.510573 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510554 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:48:42.510573 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510572 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510578 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510581 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510585 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510588 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510591 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510593 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510596 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510599 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510601 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510604 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510607 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510609 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510612 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510614 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510617 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510619 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510622 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510624 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510628 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:48:42.510656 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510632 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510635 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510638 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510642 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510645 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510648 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510650 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510652 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510655 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510658 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510662 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510665 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510669 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510672 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510674 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510677 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510679 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510681 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510684 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:48:42.511134 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510686 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510689 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510691 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510695 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510697 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510700 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510703 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510705 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510708 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510710 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510713 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510715 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510718 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510721 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510723 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510727 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510730 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510732 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510735 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:48:42.511634 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510737 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510740 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510742 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510745 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510747 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510752 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510755 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510758 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510760 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510763 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510765 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510767 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510770 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510773 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510775 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510777 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510780 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510783 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510786 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:48:42.512100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510788 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510790 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510793 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510796 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510798 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510800 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510803 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510805 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510808 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510811 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510814 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510817 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510819 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510821 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510824 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510826 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510829 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510831 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510833 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510836 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.510838 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:48:42.512576 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511590 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511598 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511600 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511603 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511606 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511608 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511612 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511616 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511618 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511622 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511625 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511627 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511630 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511632 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511635 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511637 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511640 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511643 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511645 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511648 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:48:42.513098 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511652 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511654 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511657 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511659 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511662 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511664 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511667 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511669 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511672 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511674 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511676 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511679 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511682 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511684 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511686 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511689 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511691 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511693 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511696 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511698 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:48:42.513620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511700 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511703 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511706 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511711 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511714 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511717 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511720 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511723 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511725 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511727 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511730 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511732 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511737 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511739 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511756 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511759 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511762 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511765 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511768 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:48:42.514137 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511770 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511773 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511775 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511778 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511780 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511782 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511785 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511787 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511790 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511792 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511795 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511797 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511800 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511802 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511805 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511808 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511811 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511813 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511816 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:48:42.514619 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511818 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511820 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511823 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511825 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511828 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511830 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511833 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511836 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511839 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511841 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511845 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511849 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511852 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511855 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511858 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511860 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511863 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511865 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511868 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:48:42.515092 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511870 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.511873 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512311 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512317 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512319 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512322 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512325 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512328 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512331 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512334 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512337 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512339 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512342 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512344 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512347 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512350 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512352 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512355 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512357 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512360 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:48:42.515566 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512363 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512367 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512384 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512387 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512390 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512393 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512395 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512398 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512401 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512404 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512406 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512408 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512411 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512414 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512416 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512418 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512421 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512423 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512426 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:48:42.516041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512428 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512433 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512435 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512438 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512441 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512444 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512446 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512449 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512451 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512454 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512457 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512460 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512462 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512465 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512467 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512471 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512473 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512475 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512478 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:48:42.516517 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512480 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512483 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512485 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512488 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512490 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512492 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512495 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512497 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512500 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512502 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512504 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512507 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512509 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512512 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512516 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512520 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512523 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512526 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512529 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512532 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:48:42.516991 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512535 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512537 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512540 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512542 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512545 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512548 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512550 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512554 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512557 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512559 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512562 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512564 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512566 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512569 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512571 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512574 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512576 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512578 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512581 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512583 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:48:42.517483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512586 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512589 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.512591 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512669 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512676 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512682 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512687 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512692 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512695 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512699 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512703 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512707 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512710 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512713 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512716 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512719 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512722 2580 flags.go:64] FLAG: --cgroup-root="" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512725 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512727 2580 flags.go:64] FLAG: --client-ca-file="" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512730 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512733 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512737 2580 flags.go:64] FLAG: --cluster-domain="" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512740 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512743 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 21:48:42.517955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512746 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512749 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512752 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512755 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512758 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512761 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512764 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512767 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512770 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512772 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512775 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512779 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512782 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512785 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512790 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512793 2580 flags.go:64] FLAG: --enable-server="true" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512796 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512800 2580 flags.go:64] FLAG: --event-burst="100" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512803 2580 flags.go:64] FLAG: --event-qps="50" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512806 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512809 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512812 2580 flags.go:64] FLAG: --eviction-hard="" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512816 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512819 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512822 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 21:48:42.518548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512824 2580 flags.go:64] FLAG: --eviction-soft="" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512827 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512830 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512833 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512836 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512839 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512841 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512844 2580 flags.go:64] FLAG: --feature-gates="" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512847 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512850 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512853 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512856 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512859 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512862 2580 flags.go:64] FLAG: --help="false" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512864 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-135-158.ec2.internal" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512867 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512870 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512873 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512876 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512879 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512882 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512886 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512888 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 21:48:42.519147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512891 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512894 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512897 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512900 2580 flags.go:64] FLAG: --kube-reserved="" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512903 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512906 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512909 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512912 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512915 2580 flags.go:64] FLAG: --lock-file="" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512917 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512920 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512923 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512928 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512931 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512934 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512936 2580 flags.go:64] FLAG: --logging-format="text" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512939 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512942 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512944 2580 flags.go:64] FLAG: --manifest-url="" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512947 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512952 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512955 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512958 2580 flags.go:64] FLAG: --max-pods="110" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512961 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512964 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 21:48:42.519761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512967 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512970 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512972 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512975 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512978 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512986 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512989 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512992 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512995 2580 flags.go:64] FLAG: --pod-cidr="" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.512997 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513003 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513005 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513008 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513011 2580 flags.go:64] FLAG: --port="10250" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513014 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513017 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a643542d1b6795c3" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513020 2580 flags.go:64] FLAG: --qos-reserved="" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513023 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513025 2580 flags.go:64] FLAG: --register-node="true" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513028 2580 flags.go:64] FLAG: --register-with-taints="" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513032 2580 flags.go:64] FLAG: --registry-burst="10" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513034 2580 flags.go:64] FLAG: --registry-qps="5" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513037 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513040 2580 flags.go:64] FLAG: --reserved-memory="" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513043 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 21:48:42.520364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513046 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513049 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513051 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513054 2580 flags.go:64] FLAG: --runonce="false" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513056 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513059 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513062 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513065 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513067 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513070 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513073 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513076 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513081 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513085 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513088 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513091 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513094 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513097 2580 flags.go:64] FLAG: --system-cgroups="" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513099 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513104 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513107 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513110 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513115 2580 flags.go:64] FLAG: --tls-min-version="" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513118 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513121 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 21:48:42.520984 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513123 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513126 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513129 2580 flags.go:64] FLAG: --v="2" Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513132 2580 flags.go:64] FLAG: --version="false" Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513136 2580 flags.go:64] FLAG: --vmodule="" Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513140 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.513143 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513260 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513264 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513267 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513269 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513272 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513275 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513277 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513280 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513283 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513285 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513288 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513290 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513295 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513299 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:48:42.521638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513301 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513303 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513306 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513308 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513311 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513313 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513316 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513318 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513321 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513324 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513326 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513329 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513333 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513336 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513340 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513343 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513346 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513349 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513351 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:48:42.522135 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513354 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513356 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513359 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513361 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513363 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513366 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513382 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513385 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513387 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513391 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513395 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513399 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513404 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513406 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513409 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513411 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513414 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513417 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513419 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513422 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:48:42.522597 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513424 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513426 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513436 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513439 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513441 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513444 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513446 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513448 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513451 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513453 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513456 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513458 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513461 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513463 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513466 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513468 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513471 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513473 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513476 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513478 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:48:42.523110 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513481 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513483 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513485 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513489 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513493 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513496 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513498 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513500 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513503 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513505 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513508 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513510 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513512 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513515 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513517 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513520 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513523 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513525 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513528 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513530 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:48:42.523604 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513532 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:48:42.524072 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513535 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:48:42.524072 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513537 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:48:42.524072 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513540 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:48:42.524072 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513543 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:48:42.524072 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.513545 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:48:42.524072 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.514264 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 21:48:42.524072 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.520797 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 21:48:42.524072 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.520812 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 21:48:42.524072 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.520832 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 21:48:42.524072 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.520841 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 21:48:42.524072 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520897 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:48:42.524072 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520901 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:48:42.524072 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520904 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:48:42.524072 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520907 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:48:42.524072 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520910 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520913 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520916 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520918 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520921 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520923 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520926 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520928 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520933 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520936 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520939 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520943 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520946 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520949 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520952 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520955 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520957 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520960 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520962 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:48:42.524476 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520965 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520967 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520970 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520972 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520975 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520977 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520981 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520984 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520987 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520990 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520993 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520995 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.520998 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521001 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521003 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521006 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521008 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521011 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521013 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521016 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:48:42.525051 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521018 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521021 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521023 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521026 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521028 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521030 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521034 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521038 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521042 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521045 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521047 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521050 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521052 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521055 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521057 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521059 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521062 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521064 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521068 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:48:42.525640 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521071 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521073 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521075 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521078 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521080 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521083 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521085 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521087 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521090 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521092 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521094 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521097 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521099 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521101 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521104 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521107 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521110 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521114 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521116 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521119 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:48:42.526100 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521122 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:48:42.526638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521125 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:48:42.526638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521128 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:48:42.526638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521130 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:48:42.526638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521132 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:48:42.526638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521135 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:48:42.526638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521137 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:48:42.526638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521139 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:48:42.526638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521142 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:48:42.526638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521144 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:48:42.526638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521146 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:48:42.526638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521149 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:48:42.526638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521152 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:48:42.526638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521155 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:48:42.526638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521157 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:48:42.526638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521159 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:48:42.526638 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521162 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:48:42.527060 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.521167 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 21:48:42.527060 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521282 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:48:42.527060 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521287 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:48:42.527060 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521290 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:48:42.527060 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521293 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:48:42.527060 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521295 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:48:42.527060 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521298 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:48:42.527060 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521300 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:48:42.527060 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521302 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:48:42.527060 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521305 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:48:42.527060 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521307 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:48:42.527060 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521310 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:48:42.527060 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521312 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:48:42.527060 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521315 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:48:42.527060 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521317 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521320 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521322 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521325 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521327 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521330 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521332 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521334 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521337 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521341 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521345 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521347 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521350 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521353 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521355 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521357 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521360 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521362 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521365 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:48:42.527459 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521367 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521390 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521395 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521398 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521400 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521402 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521405 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521407 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521410 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521412 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521414 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521417 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521419 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521422 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521424 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521427 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521430 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521432 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521435 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521437 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:48:42.527918 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521439 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521442 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521444 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521447 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521449 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521453 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521457 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521460 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521463 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521465 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521468 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521471 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521473 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521476 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521478 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521481 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521483 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521485 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521488 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:48:42.528416 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521490 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521493 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521496 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521499 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521502 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521505 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521508 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521511 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521514 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521516 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521519 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521522 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521524 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521527 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521529 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521532 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521535 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521537 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521540 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:48:42.528895 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521542 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:48:42.529527 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521545 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:48:42.529527 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521547 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:48:42.529527 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521550 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:48:42.529527 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521552 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:48:42.529527 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521554 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:48:42.529527 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521557 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:48:42.529527 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521559 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:48:42.529527 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:42.521562 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:48:42.529527 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.521567 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 21:48:42.529527 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.521672 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 21:48:42.529527 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.523677 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 21:48:42.529527 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.524889 2580 server.go:1019] "Starting client certificate rotation" Jul 21 21:48:42.529527 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.524988 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 21:48:42.529527 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.525022 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 21:48:42.529954 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.529767 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 21:48:42.532926 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.532906 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 21:48:42.543473 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.543453 2580 log.go:25] "Validated CRI v1 runtime API" Jul 21 21:48:42.546126 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.546111 2580 log.go:25] "Validated CRI v1 image API" Jul 21 21:48:42.546968 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.546953 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 21:48:42.549076 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.549049 2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 cdf8a934-f5a9-4a53-a877-cc54c2d1209c:/dev/nvme0n1p3 f79fb108-00d5-4eb5-9293-1717e6abb13f:/dev/nvme0n1p4] Jul 21 21:48:42.549146 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.549076 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 21:48:42.553351 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.553331 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 21:48:42.554033 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.553928 2580 manager.go:216] Machine: {Timestamp:2026-07-21 21:48:42.552705599 +0000 UTC m=+0.315939851 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099510 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec223aa5e0a34dc222e2f6caf05a6080 SystemUUID:ec223aa5-e0a3-4dc2-22e2-f6caf05a6080 BootID:63f75406-45d6-41e9-8d37-80990fa45759 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:63:80:5d:c9:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:63:80:5d:c9:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:2a:71:08:69:cc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 21:48:42.554033 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.554027 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 21:48:42.554135 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.554110 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 21:48:42.555679 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.555653 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 21:48:42.555820 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.555682 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-158.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 21:48:42.556258 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.556248 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 21:48:42.556290 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.556261 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 21:48:42.556290 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.556274 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 21:48:42.556972 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.556961 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 21:48:42.558196 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.558185 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 21:48:42.560826 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.560813 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 21 21:48:42.560939 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.560930 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 21:48:42.560979 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.560961 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 21 21:48:42.560979 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.560972 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 21:48:42.561034 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.560989 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 21:48:42.561034 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.560998 2580 kubelet.go:398] "Adding apiserver pod source" Jul 21 21:48:42.561034 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.561006 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 21:48:42.561392 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.561363 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 21:48:42.561447 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.561440 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 21:48:42.564694 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.564662 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 21:48:42.566542 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.566526 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 21:48:42.566606 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.566558 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 21:48:42.567485 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.567474 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 21:48:42.567546 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.567490 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 21:48:42.567546 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.567496 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 21:48:42.567546 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.567501 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 21:48:42.567546 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.567510 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 21:48:42.567546 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.567516 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 21:48:42.567546 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.567524 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 21:48:42.567546 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.567530 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 21:48:42.567546 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.567537 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 21:48:42.567546 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.567543 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 21:48:42.567799 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.567551 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 21:48:42.567799 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.567557 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 21:48:42.568169 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.568159 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 21:48:42.568169 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.568169 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 21:48:42.570720 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.570699 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-158.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 21:48:42.571111 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:42.571078 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 21:48:42.571157 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:42.571079 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-158.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 21:48:42.571427 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.571415 2580 server.go:1268] "Started kubelet" Jul 21 21:48:42.571526 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.571493 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 21:48:42.571580 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.571492 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 21:48:42.571580 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.571563 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 21:48:42.572258 ip-10-0-135-158 systemd[1]: Started Kubernetes Kubelet. Jul 21 21:48:42.573256 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.573105 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 21:48:42.573633 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.573619 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 21 21:48:42.576585 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:42.576559 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 21:48:42.576647 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:42.575678 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-158.ec2.internal.18c46c627afa7ec8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-158.ec2.internal,UID:ip-10-0-135-158.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-158.ec2.internal,},FirstTimestamp:2026-07-21 21:48:42.571390664 +0000 UTC m=+0.334624916,LastTimestamp:2026-07-21 21:48:42.571390664 +0000 UTC m=+0.334624916,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-158.ec2.internal,}" Jul 21 21:48:42.576903 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.576888 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l879q" Jul 21 21:48:42.578563 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.578549 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 21:48:42.578610 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.578557 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 21:48:42.578796 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:42.578662 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found" Jul 21 21:48:42.579092 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.579075 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 21:48:42.579169 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.579147 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 21:48:42.579169 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.579161 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 21:48:42.579255 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.579238 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 21 21:48:42.579255 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.579246 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 21 21:48:42.579324 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.579288 2580 factory.go:55] Registering systemd factory Jul 21 21:48:42.579324 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.579303 2580 factory.go:223] Registration of the systemd container factory successfully Jul 21 21:48:42.579520 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.579505 2580 factory.go:153] Registering CRI-O factory Jul 21 21:48:42.579520 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.579520 2580 factory.go:223] Registration of the crio container factory successfully Jul 21 21:48:42.579681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.579567 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 21:48:42.579681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.579592 2580 factory.go:103] Registering Raw factory Jul 21 21:48:42.579681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.579606 2580 manager.go:1196] Started watching for new ooms in manager Jul 21 21:48:42.580057 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.580046 2580 manager.go:349] Starting recovery of all containers Jul 21 21:48:42.582592 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:42.582566 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 21:48:42.582690 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:42.582572 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-158.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 21 21:48:42.584563 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.584529 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l879q" Jul 21 21:48:42.591252 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.591230 2580 manager.go:354] Recovery completed Jul 21 21:48:42.593106 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:42.593084 2580 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 21 21:48:42.596089 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.596076 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:42.598033 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.598017 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:42.598121 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.598052 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:42.598121 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.598067 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:42.598602 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.598591 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:42.598602 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.598596 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 21:48:42.598736 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.598605 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 21:48:42.598736 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.598619 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 21 21:48:42.599171 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.599155 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:42.599239 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.599188 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:42.599239 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.599202 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:42.599640 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.599626 2580 policy_none.go:49] "None policy: Start" Jul 21 21:48:42.599640 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.599641 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 21:48:42.599748 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.599651 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 21:48:42.600550 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.600539 2580 policy_none.go:47] "Start" Jul 21 21:48:42.643921 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.643779 2580 manager.go:337] "Starting Device Plugin manager" Jul 21 21:48:42.644025 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:42.643974 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 21:48:42.644025 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.643989 2580 server.go:87] "Starting device plugin registration server" Jul 21 21:48:42.644262 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.644248 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 21:48:42.644328 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.644264 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 21:48:42.644428 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.644363 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 21:48:42.644530 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.644491 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 21:48:42.644530 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.644505 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 21:48:42.645012 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:42.644997 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 21:48:42.645077 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:42.645037 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-158.ec2.internal\" not found" Jul 21 21:48:42.744637 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.744593 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:42.745680 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.745663 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:42.745743 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.745694 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:42.745743 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.745706 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:42.745743 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.745737 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-158.ec2.internal" Jul 21 21:48:42.754016 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.753959 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-158.ec2.internal" Jul 21 21:48:42.754016 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:42.753982 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-158.ec2.internal\": node \"ip-10-0-135-158.ec2.internal\" not found" Jul 21 21:48:42.764059 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:42.764038 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found" Jul 21 21:48:42.771290 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.771260 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 21:48:42.772584 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.772567 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 21:48:42.772677 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.772600 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 21:48:42.772677 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.772628 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 21:48:42.772677 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:42.772658 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 21:48:42.774872 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.774852 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:48:42.864618 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:42.864574 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found" Jul 21 21:48:42.873768 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.873723 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal"] Jul 21 21:48:42.873882 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.873805 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:42.875431 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.875412 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:42.875509 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.875449 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:42.875509 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.875461 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:42.876684 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.876672 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:42.876835 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.876821 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal" Jul 21 21:48:42.876884 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.876851 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:42.878588 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.878568 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:42.878588 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.878581 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:42.878588 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.878595 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:42.878742 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.878606 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:42.878742 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.878606 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:42.878742 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.878618 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:42.879872 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.879856 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal" Jul 21 21:48:42.879936 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.879887 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:42.880088 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.880072 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ba3d3cdb4f4f23180aa70b807d11a56-config\") pod \"kube-apiserver-proxy-ip-10-0-135-158.ec2.internal\" (UID: \"9ba3d3cdb4f4f23180aa70b807d11a56\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal" Jul 21 21:48:42.880156 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.880103 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/485e76d26b45188751c34edd66993253-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal\" (UID: \"485e76d26b45188751c34edd66993253\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal" Jul 21 21:48:42.880206 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.880144 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/485e76d26b45188751c34edd66993253-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal\" (UID: \"485e76d26b45188751c34edd66993253\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal" Jul 21 21:48:42.882963 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.882948 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:42.883049 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.882983 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:42.883049 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.883000 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:42.910301 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:42.910268 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-158.ec2.internal\" not found" node="ip-10-0-135-158.ec2.internal" Jul 21 21:48:42.914857 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:42.914833 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-158.ec2.internal\" not found" node="ip-10-0-135-158.ec2.internal" Jul 21 21:48:42.964681 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:42.964644 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found" Jul 21 21:48:42.980508 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.980480 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/485e76d26b45188751c34edd66993253-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal\" (UID: \"485e76d26b45188751c34edd66993253\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal" Jul 21 21:48:42.980508 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.980508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/485e76d26b45188751c34edd66993253-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal\" (UID: \"485e76d26b45188751c34edd66993253\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal" Jul 21 21:48:42.980658 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.980525 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ba3d3cdb4f4f23180aa70b807d11a56-config\") pod \"kube-apiserver-proxy-ip-10-0-135-158.ec2.internal\" (UID: \"9ba3d3cdb4f4f23180aa70b807d11a56\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal" Jul 21 21:48:42.980658 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.980585 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ba3d3cdb4f4f23180aa70b807d11a56-config\") pod \"kube-apiserver-proxy-ip-10-0-135-158.ec2.internal\" (UID: \"9ba3d3cdb4f4f23180aa70b807d11a56\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal" Jul 21 21:48:42.980658 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.980604 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/485e76d26b45188751c34edd66993253-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal\" (UID: \"485e76d26b45188751c34edd66993253\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal" Jul 21 21:48:42.980658 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:42.980626 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/485e76d26b45188751c34edd66993253-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal\" (UID: \"485e76d26b45188751c34edd66993253\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal" Jul 21 21:48:43.065710 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:43.065627 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found" Jul 21 21:48:43.166093 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:43.166054 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found" Jul 21 21:48:43.213574 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:43.213544 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal" Jul 21 21:48:43.217983 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:43.217967 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal" Jul 21 21:48:43.267057 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:43.267021 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found" Jul 21 21:48:43.367596 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:43.367521 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found" Jul 21 21:48:43.468064 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:43.468019 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found" Jul 21 21:48:43.524553 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:43.524517 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 21:48:43.525002 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:43.524674 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 21:48:43.569043 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:43.568994 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found" Jul 21 21:48:43.589127 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:43.589084 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 21:43:43 +0000 UTC" deadline="2028-03-25 17:19:08.625934095 +0000 UTC" Jul 21 21:48:43.589127 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:43.589119 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14707h30m25.036817905s" Jul 21 21:48:43.645106 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:43.645047 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:48:43.670112 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:43.670083 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found" Jul 21 21:48:43.770689 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:43.770644 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found" Jul 21 21:48:43.871531 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:43.871505 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found" Jul 21 21:48:43.874896 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:43.874842 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba3d3cdb4f4f23180aa70b807d11a56.slice/crio-af6a7f6857346c6dd3be472c48cc31744b6a58b2743a384cf03e8acd864af487 WatchSource:0}: Error finding container af6a7f6857346c6dd3be472c48cc31744b6a58b2743a384cf03e8acd864af487: Status 404 returned error can't find the container with id af6a7f6857346c6dd3be472c48cc31744b6a58b2743a384cf03e8acd864af487 Jul 21 21:48:43.875196 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:43.875179 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485e76d26b45188751c34edd66993253.slice/crio-ec576362846926bec8e5b7e2e7f18143893816557170d69a7d73af23e811cc4c WatchSource:0}: Error finding container ec576362846926bec8e5b7e2e7f18143893816557170d69a7d73af23e811cc4c: Status 404 returned error can't find the container with id ec576362846926bec8e5b7e2e7f18143893816557170d69a7d73af23e811cc4c Jul 21 21:48:43.880267 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:43.880248 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:48:43.914225 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:43.914144 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:48:43.979439 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:43.979411 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal" Jul 21 21:48:43.993006 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:43.992975 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 21:48:43.994790 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:43.994774 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal" Jul 21 21:48:44.002435 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.002416 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 21:48:44.066206 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.066167 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:48:44.562672 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.562637 2580 apiserver.go:52] "Watching apiserver" Jul 21 21:48:44.571031 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.570997 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 21:48:44.572290 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.572252 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-mxlb5","openshift-multus/network-metrics-daemon-zxm6p","openshift-network-diagnostics/network-check-target-zrk2g","openshift-network-operator/iptables-alerter-xq9jl","openshift-ovn-kubernetes/ovnkube-node-5rqkj","kube-system/global-pull-secret-syncer-fb457","kube-system/konnectivity-agent-wjhtb","kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk","openshift-cluster-node-tuning-operator/tuned-5dgn7","openshift-dns/node-resolver-fk4dn","openshift-image-registry/node-ca-jdcql","openshift-multus/multus-additional-cni-plugins-t6jrl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal"] Jul 21 21:48:44.574277 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.574245 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wjhtb" Jul 21 21:48:44.575906 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.575887 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.577013 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.576994 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:48:44.577111 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:44.577061 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:48:44.577430 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.577414 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 21:48:44.577516 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.577466 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 21:48:44.577573 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.577542 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r87v9\"" Jul 21 21:48:44.578259 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.578243 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 21:48:44.578629 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.578610 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 21:48:44.578970 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.578938 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 21:48:44.579148 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.579133 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9qb2v\"" Jul 21 21:48:44.579343 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.579328 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 21:48:44.580209 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.580166 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 21:48:44.581240 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.580668 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.581240 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.580851 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xq9jl" Jul 21 21:48:44.581921 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.581732 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 21:48:44.582411 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.582283 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:44.582411 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:44.582352 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:48:44.583359 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.583182 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pjv9w\"" Jul 21 21:48:44.583841 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.583824 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 21:48:44.584039 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.584022 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:48:44.585126 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.584271 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 21:48:44.585126 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.584424 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 21:48:44.585126 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.584713 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.585126 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.584818 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 21:48:44.589867 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.586209 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 21:48:44.589867 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.586556 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.589867 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.588065 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-blt9n\"" Jul 21 21:48:44.589867 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.588104 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 21:48:44.589867 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.588882 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 21:48:44.589867 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.589204 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 21:48:44.590655 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.590629 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 21:48:44.590794 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.590695 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 21:48:44.591353 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.591333 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d856x\"" Jul 21 21:48:44.591430 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.591384 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08eea2bf-67da-433f-9897-4f5ea47f5fe7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.591513 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.591453 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz2qm\" (UniqueName: \"kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm\") pod \"network-check-target-zrk2g\" (UID: \"1b7dcb72-e10b-4f88-be68-79d8b3b97f87\") " pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:48:44.591513 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.591502 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7935139c-997c-4f11-83b9-cca9a0053e65-iptables-alerter-script\") pod \"iptables-alerter-xq9jl\" (UID: \"7935139c-997c-4f11-83b9-cca9a0053e65\") " pod="openshift-network-operator/iptables-alerter-xq9jl" Jul 21 21:48:44.591622 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.591534 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08eea2bf-67da-433f-9897-4f5ea47f5fe7-system-cni-dir\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.591622 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.591568 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-kubelet\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.591622 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.591603 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.591818 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.591797 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8619db3e-9222-4014-829d-7fe3e363d332-ovnkube-script-lib\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.591883 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.591842 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7935139c-997c-4f11-83b9-cca9a0053e65-host-slash\") pod \"iptables-alerter-xq9jl\" (UID: \"7935139c-997c-4f11-83b9-cca9a0053e65\") " pod="openshift-network-operator/iptables-alerter-xq9jl" Jul 21 21:48:44.591944 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.591879 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwg6d\" (UniqueName: \"kubernetes.io/projected/7935139c-997c-4f11-83b9-cca9a0053e65-kube-api-access-lwg6d\") pod \"iptables-alerter-xq9jl\" (UID: \"7935139c-997c-4f11-83b9-cca9a0053e65\") " pod="openshift-network-operator/iptables-alerter-xq9jl" Jul 21 21:48:44.591944 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.591904 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-run-ovn\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.591944 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.591933 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-log-socket\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.592096 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.591962 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08eea2bf-67da-433f-9897-4f5ea47f5fe7-cni-binary-copy\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.592096 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.591992 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 21:48:44.592096 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.592008 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08eea2bf-67da-433f-9897-4f5ea47f5fe7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.592580 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.592557 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08eea2bf-67da-433f-9897-4f5ea47f5fe7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.592651 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.592608 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vmdzp\"" Jul 21 21:48:44.592651 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.592606 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-run-netns\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.592756 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.592643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-run-systemd\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.592756 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.592677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-node-log\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.592756 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.592720 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-run-ovn-kubernetes\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.592912 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.592759 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-slash\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.592912 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.592793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-run-openvswitch\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.592912 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.592825 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8619db3e-9222-4014-829d-7fe3e363d332-ovn-node-metrics-cert\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.593067 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.592980 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/08eea2bf-67da-433f-9897-4f5ea47f5fe7-os-release\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.593067 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.593015 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qp2\" (UniqueName: \"kubernetes.io/projected/08eea2bf-67da-433f-9897-4f5ea47f5fe7-kube-api-access-88qp2\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.593067 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.593054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-systemd-units\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.593219 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.593086 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-etc-openvswitch\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.593219 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.593116 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-cni-netd\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.593219 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.593173 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8619db3e-9222-4014-829d-7fe3e363d332-ovnkube-config\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.593371 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.593315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8619db3e-9222-4014-829d-7fe3e363d332-env-overrides\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.593371 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.593344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wl6j\" (UniqueName: \"kubernetes.io/projected/8619db3e-9222-4014-829d-7fe3e363d332-kube-api-access-6wl6j\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.593505 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.593431 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0c0ed0aa-b6e4-4b63-8ab3-378c8e1024aa-konnectivity-ca\") pod \"konnectivity-agent-wjhtb\" (UID: \"0c0ed0aa-b6e4-4b63-8ab3-378c8e1024aa\") " pod="kube-system/konnectivity-agent-wjhtb" Jul 21 21:48:44.593562 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.593501 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08eea2bf-67da-433f-9897-4f5ea47f5fe7-cnibin\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.593744 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.593725 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0c0ed0aa-b6e4-4b63-8ab3-378c8e1024aa-agent-certs\") pod \"konnectivity-agent-wjhtb\" (UID: \"0c0ed0aa-b6e4-4b63-8ab3-378c8e1024aa\") " pod="kube-system/konnectivity-agent-wjhtb" Jul 21 21:48:44.594023 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.593991 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-var-lib-openvswitch\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.594091 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.594044 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-cni-bin\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.595777 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.594728 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 21:48:44.595883 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.595784 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:48:44.597460 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.597212 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 21:48:44.602294 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:44.601575 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:48:44.603188 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.603166 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.605037 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.604645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fk4dn" Jul 21 21:48:44.605955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.605767 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:48:44.606254 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.606226 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f84mv\"" Jul 21 21:48:44.606351 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.606299 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 21:48:44.606782 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.606757 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jdcql" Jul 21 21:48:44.607343 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.607324 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 21:48:44.607499 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.607476 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 21:48:44.607582 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.607568 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4twls\"" Jul 21 21:48:44.610502 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.610482 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bngqb" Jul 21 21:48:44.610631 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.610614 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 21:48:44.610869 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.610855 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 21:48:44.611434 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.611036 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8x72r\"" Jul 21 21:48:44.611434 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.611271 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 21:48:44.620392 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.620368 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bngqb" Jul 21 21:48:44.681016 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.680982 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 21:48:44.694524 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.694484 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-host-var-lib-cni-bin\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.694675 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.694530 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-modprobe-d\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.694675 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.694557 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfc9f\" (UniqueName: \"kubernetes.io/projected/2989142e-cd19-4991-85d2-6c75e18ce1ae-kube-api-access-wfc9f\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.694675 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.694606 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6c5658d-ed28-4bef-ada7-ab4746a7a346-host\") pod \"node-ca-jdcql\" (UID: \"d6c5658d-ed28-4bef-ada7-ab4746a7a346\") " pod="openshift-image-registry/node-ca-jdcql" Jul 21 21:48:44.694824 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.694668 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/08eea2bf-67da-433f-9897-4f5ea47f5fe7-os-release\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.694824 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.694703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qp2\" (UniqueName: \"kubernetes.io/projected/08eea2bf-67da-433f-9897-4f5ea47f5fe7-kube-api-access-88qp2\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.694824 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.694731 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-systemd-units\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.694824 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.694757 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-cni-netd\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.694824 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.694784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8619db3e-9222-4014-829d-7fe3e363d332-ovnkube-config\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.695065 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.694817 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8619db3e-9222-4014-829d-7fe3e363d332-env-overrides\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.695065 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.694860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wl6j\" (UniqueName: \"kubernetes.io/projected/8619db3e-9222-4014-829d-7fe3e363d332-kube-api-access-6wl6j\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.695065 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.694869 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/08eea2bf-67da-433f-9897-4f5ea47f5fe7-os-release\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.695065 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.694907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5daa5430-2084-48d2-a008-6305a894ea65-hosts-file\") pod \"node-resolver-fk4dn\" (UID: \"5daa5430-2084-48d2-a008-6305a894ea65\") " pod="openshift-dns/node-resolver-fk4dn" Jul 21 21:48:44.695065 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.694934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0c0ed0aa-b6e4-4b63-8ab3-378c8e1024aa-konnectivity-ca\") pod \"konnectivity-agent-wjhtb\" (UID: \"0c0ed0aa-b6e4-4b63-8ab3-378c8e1024aa\") " pod="kube-system/konnectivity-agent-wjhtb" Jul 21 21:48:44.695065 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.694940 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-cni-netd\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.695065 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.694969 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-system-cni-dir\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.695065 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695007 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-systemd\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.695065 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695029 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-sys\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.695065 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695062 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-host\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.695567 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0c0ed0aa-b6e4-4b63-8ab3-378c8e1024aa-agent-certs\") pod \"konnectivity-agent-wjhtb\" (UID: \"0c0ed0aa-b6e4-4b63-8ab3-378c8e1024aa\") " pod="kube-system/konnectivity-agent-wjhtb" Jul 21 21:48:44.695567 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-cni-bin\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.695567 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695179 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3de97b9-626a-4516-9749-94ff62f60273-kubelet-config\") pod \"global-pull-secret-syncer-fb457\" (UID: \"b3de97b9-626a-4516-9749-94ff62f60273\") " pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:44.695567 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7935139c-997c-4f11-83b9-cca9a0053e65-iptables-alerter-script\") pod \"iptables-alerter-xq9jl\" (UID: \"7935139c-997c-4f11-83b9-cca9a0053e65\") " pod="openshift-network-operator/iptables-alerter-xq9jl" Jul 21 21:48:44.695567 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695304 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-multus-daemon-config\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.695567 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tpqw\" (UniqueName: \"kubernetes.io/projected/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-kube-api-access-5tpqw\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.695567 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695342 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08eea2bf-67da-433f-9897-4f5ea47f5fe7-system-cni-dir\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.695567 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-kubelet\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.695567 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.695567 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695423 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-host-run-multus-certs\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.695567 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695451 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-systemd-units\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.695567 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.695567 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695501 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7935139c-997c-4f11-83b9-cca9a0053e65-host-slash\") pod \"iptables-alerter-xq9jl\" (UID: \"7935139c-997c-4f11-83b9-cca9a0053e65\") " pod="openshift-network-operator/iptables-alerter-xq9jl" Jul 21 21:48:44.695567 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695562 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-kubelet\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.696307 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695569 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-cni-bin\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.696307 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-log-socket\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.696307 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695665 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-host-run-netns\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.696307 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695764 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7935139c-997c-4f11-83b9-cca9a0053e65-host-slash\") pod \"iptables-alerter-xq9jl\" (UID: \"7935139c-997c-4f11-83b9-cca9a0053e65\") " pod="openshift-network-operator/iptables-alerter-xq9jl" Jul 21 21:48:44.696307 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695794 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08eea2bf-67da-433f-9897-4f5ea47f5fe7-system-cni-dir\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.696307 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695879 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-host-var-lib-kubelet\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.696307 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696276 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0c0ed0aa-b6e4-4b63-8ab3-378c8e1024aa-konnectivity-ca\") pod \"konnectivity-agent-wjhtb\" (UID: \"0c0ed0aa-b6e4-4b63-8ab3-378c8e1024aa\") " pod="kube-system/konnectivity-agent-wjhtb" Jul 21 21:48:44.696687 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696359 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 21:48:44.696687 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696371 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8619db3e-9222-4014-829d-7fe3e363d332-env-overrides\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.696687 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696476 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.696687 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.695994 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-log-socket\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.696687 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696561 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-registration-dir\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.696687 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696589 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d6c5658d-ed28-4bef-ada7-ab4746a7a346-serviceca\") pod \"node-ca-jdcql\" (UID: \"d6c5658d-ed28-4bef-ada7-ab4746a7a346\") " pod="openshift-image-registry/node-ca-jdcql" Jul 21 21:48:44.696687 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08eea2bf-67da-433f-9897-4f5ea47f5fe7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.696687 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696636 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-cni-binary-copy\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.696687 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696644 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7935139c-997c-4f11-83b9-cca9a0053e65-iptables-alerter-script\") pod \"iptables-alerter-xq9jl\" (UID: \"7935139c-997c-4f11-83b9-cca9a0053e65\") " pod="openshift-network-operator/iptables-alerter-xq9jl" Jul 21 21:48:44.696687 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret\") pod \"global-pull-secret-syncer-fb457\" (UID: \"b3de97b9-626a-4516-9749-94ff62f60273\") " pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:44.696687 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696682 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-device-dir\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.697157 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-run\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.697157 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696802 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-run-openvswitch\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.697157 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696851 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-cnibin\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.697157 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-multus-conf-dir\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.697157 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696900 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-sys-fs\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.697157 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696900 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-run-openvswitch\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.697157 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696922 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpkd5\" (UniqueName: \"kubernetes.io/projected/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-kube-api-access-zpkd5\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.697157 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696941 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2989142e-cd19-4991-85d2-6c75e18ce1ae-tmp\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.697157 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.696960 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8619db3e-9222-4014-829d-7fe3e363d332-ovnkube-config\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.697157 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697009 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-etc-openvswitch\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.697157 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697047 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-os-release\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.697157 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697060 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-etc-openvswitch\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.697157 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697073 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-multus-socket-dir-parent\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.697157 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-host-run-k8s-cni-cncf-io\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.697681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697178 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-hostroot\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.697681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697195 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08eea2bf-67da-433f-9897-4f5ea47f5fe7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.697681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697200 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sf8p\" (UniqueName: \"kubernetes.io/projected/5daa5430-2084-48d2-a008-6305a894ea65-kube-api-access-2sf8p\") pod \"node-resolver-fk4dn\" (UID: \"5daa5430-2084-48d2-a008-6305a894ea65\") " pod="openshift-dns/node-resolver-fk4dn" Jul 21 21:48:44.697681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697224 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs\") pod \"network-metrics-daemon-zxm6p\" (UID: \"052e4590-9613-4a30-b87c-fe56d7a11e9a\") " pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:48:44.697681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697242 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-kubernetes\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.697681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08eea2bf-67da-433f-9897-4f5ea47f5fe7-cnibin\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.697681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697300 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-etc-selinux\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.697681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697314 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08eea2bf-67da-433f-9897-4f5ea47f5fe7-cnibin\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.697681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgq72\" (UniqueName: \"kubernetes.io/projected/052e4590-9613-4a30-b87c-fe56d7a11e9a-kube-api-access-zgq72\") pod \"network-metrics-daemon-zxm6p\" (UID: \"052e4590-9613-4a30-b87c-fe56d7a11e9a\") " pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:48:44.697681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697359 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-sysctl-d\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.697681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697413 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-var-lib-kubelet\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.697681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697526 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-tuned\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.697681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-var-lib-openvswitch\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.697681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-etc-kubernetes\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.697681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697681 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-sysconfig\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.698294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697706 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n9f5\" (UniqueName: \"kubernetes.io/projected/d6c5658d-ed28-4bef-ada7-ab4746a7a346-kube-api-access-7n9f5\") pod \"node-ca-jdcql\" (UID: \"d6c5658d-ed28-4bef-ada7-ab4746a7a346\") " pod="openshift-image-registry/node-ca-jdcql" Jul 21 21:48:44.698294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697727 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-var-lib-openvswitch\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.698294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08eea2bf-67da-433f-9897-4f5ea47f5fe7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.698294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697758 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2qm\" (UniqueName: \"kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm\") pod \"network-check-target-zrk2g\" (UID: \"1b7dcb72-e10b-4f88-be68-79d8b3b97f87\") " pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:48:44.698294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697816 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-host-var-lib-cni-multus\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.698294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5daa5430-2084-48d2-a008-6305a894ea65-tmp-dir\") pod \"node-resolver-fk4dn\" (UID: \"5daa5430-2084-48d2-a008-6305a894ea65\") " pod="openshift-dns/node-resolver-fk4dn" Jul 21 21:48:44.698294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697877 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-socket-dir\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.698294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697898 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-sysctl-conf\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.698294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697931 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8619db3e-9222-4014-829d-7fe3e363d332-ovnkube-script-lib\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.698294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697954 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-multus-cni-dir\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.698294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.697982 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwg6d\" (UniqueName: \"kubernetes.io/projected/7935139c-997c-4f11-83b9-cca9a0053e65-kube-api-access-lwg6d\") pod \"iptables-alerter-xq9jl\" (UID: \"7935139c-997c-4f11-83b9-cca9a0053e65\") " pod="openshift-network-operator/iptables-alerter-xq9jl" Jul 21 21:48:44.698294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698008 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-run-ovn\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.698294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698038 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3de97b9-626a-4516-9749-94ff62f60273-dbus\") pod \"global-pull-secret-syncer-fb457\" (UID: \"b3de97b9-626a-4516-9749-94ff62f60273\") " pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:44.698294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698061 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-run-ovn\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.698294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698064 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-lib-modules\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.698294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698118 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08eea2bf-67da-433f-9897-4f5ea47f5fe7-cni-binary-copy\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.698294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08eea2bf-67da-433f-9897-4f5ea47f5fe7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.698924 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-run-netns\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.698924 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-run-systemd\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.698924 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698214 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08eea2bf-67da-433f-9897-4f5ea47f5fe7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.698924 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698238 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-node-log\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.698924 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698271 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-run-ovn-kubernetes\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.698924 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698286 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-run-netns\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.698924 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698302 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-slash\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.698924 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698330 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-node-log\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.698924 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698328 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8619db3e-9222-4014-829d-7fe3e363d332-ovn-node-metrics-cert\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.698924 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698375 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-run-ovn-kubernetes\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.698924 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698411 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08eea2bf-67da-433f-9897-4f5ea47f5fe7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.698924 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698435 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-host-slash\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.698924 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698380 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8619db3e-9222-4014-829d-7fe3e363d332-run-systemd\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.698924 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698508 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8619db3e-9222-4014-829d-7fe3e363d332-ovnkube-script-lib\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.698924 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.698644 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08eea2bf-67da-433f-9897-4f5ea47f5fe7-cni-binary-copy\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.700549 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.700526 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0c0ed0aa-b6e4-4b63-8ab3-378c8e1024aa-agent-certs\") pod \"konnectivity-agent-wjhtb\" (UID: \"0c0ed0aa-b6e4-4b63-8ab3-378c8e1024aa\") " pod="kube-system/konnectivity-agent-wjhtb" Jul 21 21:48:44.700941 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.700918 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8619db3e-9222-4014-829d-7fe3e363d332-ovn-node-metrics-cert\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.706322 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.706293 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wl6j\" (UniqueName: \"kubernetes.io/projected/8619db3e-9222-4014-829d-7fe3e363d332-kube-api-access-6wl6j\") pod \"ovnkube-node-5rqkj\" (UID: \"8619db3e-9222-4014-829d-7fe3e363d332\") " pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.706819 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.706792 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qp2\" (UniqueName: \"kubernetes.io/projected/08eea2bf-67da-433f-9897-4f5ea47f5fe7-kube-api-access-88qp2\") pod \"multus-additional-cni-plugins-t6jrl\" (UID: \"08eea2bf-67da-433f-9897-4f5ea47f5fe7\") " pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.706941 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:44.706924 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:48:44.706993 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:44.706950 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:48:44.706993 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:44.706968 2580 projected.go:196] Error preparing data for projected volume kube-api-access-bz2qm for pod openshift-network-diagnostics/network-check-target-zrk2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:44.707055 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:44.707033 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm podName:1b7dcb72-e10b-4f88-be68-79d8b3b97f87 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:45.206998748 +0000 UTC m=+2.970232989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bz2qm" (UniqueName: "kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm") pod "network-check-target-zrk2g" (UID: "1b7dcb72-e10b-4f88-be68-79d8b3b97f87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:44.707235 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.707219 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwg6d\" (UniqueName: \"kubernetes.io/projected/7935139c-997c-4f11-83b9-cca9a0053e65-kube-api-access-lwg6d\") pod \"iptables-alerter-xq9jl\" (UID: \"7935139c-997c-4f11-83b9-cca9a0053e65\") " pod="openshift-network-operator/iptables-alerter-xq9jl" Jul 21 21:48:44.757048 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.756997 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:48:44.778226 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.778167 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal" event={"ID":"485e76d26b45188751c34edd66993253","Type":"ContainerStarted","Data":"ec576362846926bec8e5b7e2e7f18143893816557170d69a7d73af23e811cc4c"} Jul 21 21:48:44.780370 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.780335 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal" event={"ID":"9ba3d3cdb4f4f23180aa70b807d11a56","Type":"ContainerStarted","Data":"af6a7f6857346c6dd3be472c48cc31744b6a58b2743a384cf03e8acd864af487"} Jul 21 21:48:44.798681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.798653 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-cni-binary-copy\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.798840 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.798688 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret\") pod \"global-pull-secret-syncer-fb457\" (UID: \"b3de97b9-626a-4516-9749-94ff62f60273\") " pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:44.798840 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.798712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-device-dir\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.798840 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.798775 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-device-dir\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.798840 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.798811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-run\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.798840 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:44.798822 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:44.798840 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.798834 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-cnibin\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.799139 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.798858 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-multus-conf-dir\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.799139 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.798882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-sys-fs\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.799139 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.798911 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpkd5\" (UniqueName: \"kubernetes.io/projected/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-kube-api-access-zpkd5\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.799139 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:44.798929 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret podName:b3de97b9-626a-4516-9749-94ff62f60273 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:45.298912 +0000 UTC m=+3.062146240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret") pod "global-pull-secret-syncer-fb457" (UID: "b3de97b9-626a-4516-9749-94ff62f60273") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:44.799139 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.798961 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-run\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.799139 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799013 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-multus-conf-dir\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.799139 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799053 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-cnibin\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.799139 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799069 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2989142e-cd19-4991-85d2-6c75e18ce1ae-tmp\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.799139 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-os-release\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.799139 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799119 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-sys-fs\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.799139 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799129 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-multus-socket-dir-parent\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.799719 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-host-run-k8s-cni-cncf-io\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.799719 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-hostroot\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.799719 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799206 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sf8p\" (UniqueName: \"kubernetes.io/projected/5daa5430-2084-48d2-a008-6305a894ea65-kube-api-access-2sf8p\") pod \"node-resolver-fk4dn\" (UID: \"5daa5430-2084-48d2-a008-6305a894ea65\") " pod="openshift-dns/node-resolver-fk4dn" Jul 21 21:48:44.799719 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799214 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-host-run-k8s-cni-cncf-io\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.799719 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799181 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-os-release\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.799719 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799233 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs\") pod \"network-metrics-daemon-zxm6p\" (UID: \"052e4590-9613-4a30-b87c-fe56d7a11e9a\") " pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:48:44.799719 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799258 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-hostroot\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.799719 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799260 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-kubernetes\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.799719 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-etc-selinux\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.799719 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799314 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-kubernetes\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.799719 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799330 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgq72\" (UniqueName: \"kubernetes.io/projected/052e4590-9613-4a30-b87c-fe56d7a11e9a-kube-api-access-zgq72\") pod \"network-metrics-daemon-zxm6p\" (UID: \"052e4590-9613-4a30-b87c-fe56d7a11e9a\") " pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:48:44.799719 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799218 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-multus-socket-dir-parent\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.799719 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:44.799430 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:44.799719 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799413 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-cni-binary-copy\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.799719 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799491 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-sysctl-d\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.799719 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:44.799500 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs podName:052e4590-9613-4a30-b87c-fe56d7a11e9a nodeName:}" failed. No retries permitted until 2026-07-21 21:48:45.299480935 +0000 UTC m=+3.062715181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs") pod "network-metrics-daemon-zxm6p" (UID: "052e4590-9613-4a30-b87c-fe56d7a11e9a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:44.799719 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799496 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-etc-selinux\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.800426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-var-lib-kubelet\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.800426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-tuned\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.800426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-etc-kubernetes\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.800426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799586 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-sysconfig\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.800426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799603 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-var-lib-kubelet\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.800426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799614 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-sysctl-d\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.800426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n9f5\" (UniqueName: \"kubernetes.io/projected/d6c5658d-ed28-4bef-ada7-ab4746a7a346-kube-api-access-7n9f5\") pod \"node-ca-jdcql\" (UID: \"d6c5658d-ed28-4bef-ada7-ab4746a7a346\") " pod="openshift-image-registry/node-ca-jdcql" Jul 21 21:48:44.800426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799655 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-etc-kubernetes\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.800426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799674 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-host-var-lib-cni-multus\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.800426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5daa5430-2084-48d2-a008-6305a894ea65-tmp-dir\") pod \"node-resolver-fk4dn\" (UID: \"5daa5430-2084-48d2-a008-6305a894ea65\") " pod="openshift-dns/node-resolver-fk4dn" Jul 21 21:48:44.800426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799713 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-host-var-lib-cni-multus\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.800426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799728 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-socket-dir\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.800426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799676 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-sysconfig\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.800426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-sysctl-conf\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.800426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799847 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-socket-dir\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.800426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799969 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-multus-cni-dir\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.800426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799976 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-sysctl-conf\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.800426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.799998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3de97b9-626a-4516-9749-94ff62f60273-dbus\") pod \"global-pull-secret-syncer-fb457\" (UID: \"b3de97b9-626a-4516-9749-94ff62f60273\") " pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:44.801042 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-lib-modules\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.801042 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800038 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5daa5430-2084-48d2-a008-6305a894ea65-tmp-dir\") pod \"node-resolver-fk4dn\" (UID: \"5daa5430-2084-48d2-a008-6305a894ea65\") " pod="openshift-dns/node-resolver-fk4dn" Jul 21 21:48:44.801042 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800058 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-host-var-lib-cni-bin\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.801042 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800084 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-modprobe-d\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.801042 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800109 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfc9f\" (UniqueName: \"kubernetes.io/projected/2989142e-cd19-4991-85d2-6c75e18ce1ae-kube-api-access-wfc9f\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.801042 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800116 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-multus-cni-dir\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.801042 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800165 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-lib-modules\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.801042 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800170 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6c5658d-ed28-4bef-ada7-ab4746a7a346-host\") pod \"node-ca-jdcql\" (UID: \"d6c5658d-ed28-4bef-ada7-ab4746a7a346\") " pod="openshift-image-registry/node-ca-jdcql" Jul 21 21:48:44.801042 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6c5658d-ed28-4bef-ada7-ab4746a7a346-host\") pod \"node-ca-jdcql\" (UID: \"d6c5658d-ed28-4bef-ada7-ab4746a7a346\") " pod="openshift-image-registry/node-ca-jdcql" Jul 21 21:48:44.801042 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800219 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3de97b9-626a-4516-9749-94ff62f60273-dbus\") pod \"global-pull-secret-syncer-fb457\" (UID: \"b3de97b9-626a-4516-9749-94ff62f60273\") " pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:44.801042 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800270 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-modprobe-d\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.801042 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800205 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-host-var-lib-cni-bin\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.801042 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5daa5430-2084-48d2-a008-6305a894ea65-hosts-file\") pod \"node-resolver-fk4dn\" (UID: \"5daa5430-2084-48d2-a008-6305a894ea65\") " pod="openshift-dns/node-resolver-fk4dn" Jul 21 21:48:44.801042 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-system-cni-dir\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.801042 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-systemd\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.801042 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-sys\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.801042 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-system-cni-dir\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.801042 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800556 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-host\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.801834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800574 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-systemd\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.801834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800504 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5daa5430-2084-48d2-a008-6305a894ea65-hosts-file\") pod \"node-resolver-fk4dn\" (UID: \"5daa5430-2084-48d2-a008-6305a894ea65\") " pod="openshift-dns/node-resolver-fk4dn" Jul 21 21:48:44.801834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-sys\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.801834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800591 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3de97b9-626a-4516-9749-94ff62f60273-kubelet-config\") pod \"global-pull-secret-syncer-fb457\" (UID: \"b3de97b9-626a-4516-9749-94ff62f60273\") " pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:44.801834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800636 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-multus-daemon-config\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.801834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800641 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3de97b9-626a-4516-9749-94ff62f60273-kubelet-config\") pod \"global-pull-secret-syncer-fb457\" (UID: \"b3de97b9-626a-4516-9749-94ff62f60273\") " pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:44.801834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800645 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2989142e-cd19-4991-85d2-6c75e18ce1ae-host\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.801834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800673 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tpqw\" (UniqueName: \"kubernetes.io/projected/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-kube-api-access-5tpqw\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.801834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800705 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-host-run-multus-certs\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.801834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.801834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800785 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.801834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800822 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-host-run-multus-certs\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.801834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800958 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-host-run-netns\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.801834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.800985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-host-var-lib-kubelet\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.801834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.801010 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-registration-dir\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.801834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.801018 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-host-run-netns\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.801834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.801074 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-host-var-lib-kubelet\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.801834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.801124 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-registration-dir\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.802774 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.801035 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d6c5658d-ed28-4bef-ada7-ab4746a7a346-serviceca\") pod \"node-ca-jdcql\" (UID: \"d6c5658d-ed28-4bef-ada7-ab4746a7a346\") " pod="openshift-image-registry/node-ca-jdcql" Jul 21 21:48:44.802774 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.801120 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-multus-daemon-config\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.802774 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.801541 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d6c5658d-ed28-4bef-ada7-ab4746a7a346-serviceca\") pod \"node-ca-jdcql\" (UID: \"d6c5658d-ed28-4bef-ada7-ab4746a7a346\") " pod="openshift-image-registry/node-ca-jdcql" Jul 21 21:48:44.802774 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.801919 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2989142e-cd19-4991-85d2-6c75e18ce1ae-tmp\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.802774 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.802108 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2989142e-cd19-4991-85d2-6c75e18ce1ae-etc-tuned\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.810912 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.810888 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpkd5\" (UniqueName: \"kubernetes.io/projected/4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b-kube-api-access-zpkd5\") pod \"aws-ebs-csi-driver-node-przmk\" (UID: \"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.811757 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.811734 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfc9f\" (UniqueName: \"kubernetes.io/projected/2989142e-cd19-4991-85d2-6c75e18ce1ae-kube-api-access-wfc9f\") pod \"tuned-5dgn7\" (UID: \"2989142e-cd19-4991-85d2-6c75e18ce1ae\") " pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.811875 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.811841 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tpqw\" (UniqueName: \"kubernetes.io/projected/6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96-kube-api-access-5tpqw\") pod \"multus-mxlb5\" (UID: \"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96\") " pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.811918 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.811881 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgq72\" (UniqueName: \"kubernetes.io/projected/052e4590-9613-4a30-b87c-fe56d7a11e9a-kube-api-access-zgq72\") pod \"network-metrics-daemon-zxm6p\" (UID: \"052e4590-9613-4a30-b87c-fe56d7a11e9a\") " pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:48:44.813380 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.813319 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sf8p\" (UniqueName: \"kubernetes.io/projected/5daa5430-2084-48d2-a008-6305a894ea65-kube-api-access-2sf8p\") pod \"node-resolver-fk4dn\" (UID: \"5daa5430-2084-48d2-a008-6305a894ea65\") " pod="openshift-dns/node-resolver-fk4dn" Jul 21 21:48:44.813815 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.813797 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n9f5\" (UniqueName: \"kubernetes.io/projected/d6c5658d-ed28-4bef-ada7-ab4746a7a346-kube-api-access-7n9f5\") pod \"node-ca-jdcql\" (UID: \"d6c5658d-ed28-4bef-ada7-ab4746a7a346\") " pod="openshift-image-registry/node-ca-jdcql" Jul 21 21:48:44.890545 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.890508 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wjhtb" Jul 21 21:48:44.904805 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.904778 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t6jrl" Jul 21 21:48:44.918227 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.918201 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:48:44.926256 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.926231 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xq9jl" Jul 21 21:48:44.935159 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.935136 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" Jul 21 21:48:44.942823 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.942803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mxlb5" Jul 21 21:48:44.948553 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.948535 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" Jul 21 21:48:44.958475 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.958455 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fk4dn" Jul 21 21:48:44.966276 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:44.966249 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jdcql" Jul 21 21:48:45.305696 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:45.305604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2qm\" (UniqueName: \"kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm\") pod \"network-check-target-zrk2g\" (UID: \"1b7dcb72-e10b-4f88-be68-79d8b3b97f87\") " pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:48:45.305844 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:45.305693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret\") pod \"global-pull-secret-syncer-fb457\" (UID: \"b3de97b9-626a-4516-9749-94ff62f60273\") " pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:45.305844 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:45.305736 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs\") pod \"network-metrics-daemon-zxm6p\" (UID: \"052e4590-9613-4a30-b87c-fe56d7a11e9a\") " pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:48:45.305844 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:45.305750 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:48:45.305844 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:45.305781 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:48:45.305844 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:45.305797 2580 projected.go:196] Error preparing data for projected volume kube-api-access-bz2qm for pod openshift-network-diagnostics/network-check-target-zrk2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:45.305844 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:45.305838 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:45.305844 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:45.305846 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:45.306195 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:45.305873 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm podName:1b7dcb72-e10b-4f88-be68-79d8b3b97f87 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:46.305852075 +0000 UTC m=+4.069086337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bz2qm" (UniqueName: "kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm") pod "network-check-target-zrk2g" (UID: "1b7dcb72-e10b-4f88-be68-79d8b3b97f87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:45.306195 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:45.305899 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs podName:052e4590-9613-4a30-b87c-fe56d7a11e9a nodeName:}" failed. No retries permitted until 2026-07-21 21:48:46.305883202 +0000 UTC m=+4.069117446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs") pod "network-metrics-daemon-zxm6p" (UID: "052e4590-9613-4a30-b87c-fe56d7a11e9a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:45.306195 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:45.305923 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret podName:b3de97b9-626a-4516-9749-94ff62f60273 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:46.305908177 +0000 UTC m=+4.069142425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret") pod "global-pull-secret-syncer-fb457" (UID: "b3de97b9-626a-4516-9749-94ff62f60273") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:45.624043 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:45.623913 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 21:43:45 +0000 UTC" deadline="2028-03-23 18:30:10.868240095 +0000 UTC" Jul 21 21:48:45.624043 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:45.623986 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14660h41m25.244257903s" Jul 21 21:48:45.773588 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:45.773541 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:45.773761 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:45.773682 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:48:45.832980 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:45.832937 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7935139c_997c_4f11_83b9_cca9a0053e65.slice/crio-8031d20ed25796194cd610b236cf6f83a3b33e8cfc372325f7a7deb5234e7fdb WatchSource:0}: Error finding container 8031d20ed25796194cd610b236cf6f83a3b33e8cfc372325f7a7deb5234e7fdb: Status 404 returned error can't find the container with id 8031d20ed25796194cd610b236cf6f83a3b33e8cfc372325f7a7deb5234e7fdb Jul 21 21:48:45.838498 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:45.838467 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0ed0aa_b6e4_4b63_8ab3_378c8e1024aa.slice/crio-4a0cefb7c47084ccbf1e0aed20c65c68470f9744a4de28cd23b0f4184a6f350d WatchSource:0}: Error finding container 4a0cefb7c47084ccbf1e0aed20c65c68470f9744a4de28cd23b0f4184a6f350d: Status 404 returned error can't find the container with id 4a0cefb7c47084ccbf1e0aed20c65c68470f9744a4de28cd23b0f4184a6f350d Jul 21 21:48:45.839128 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:45.839105 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2989142e_cd19_4991_85d2_6c75e18ce1ae.slice/crio-253bab74830f1db291d1241a82471143b5e0eebedbadeeeb76b2f79ad1baae16 WatchSource:0}: Error finding container 253bab74830f1db291d1241a82471143b5e0eebedbadeeeb76b2f79ad1baae16: Status 404 returned error can't find the container with id 253bab74830f1db291d1241a82471143b5e0eebedbadeeeb76b2f79ad1baae16 Jul 21 21:48:45.840598 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:45.840486 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a67a0b6_b146_4679_8ad7_9d1c7fdd7d96.slice/crio-1de1b5e2c7de7bf09381b93e43f919125a62607a60aea95a364827bdae82ed7d WatchSource:0}: Error finding container 1de1b5e2c7de7bf09381b93e43f919125a62607a60aea95a364827bdae82ed7d: Status 404 returned error can't find the container with id 1de1b5e2c7de7bf09381b93e43f919125a62607a60aea95a364827bdae82ed7d Jul 21 21:48:45.841620 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:45.841597 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08eea2bf_67da_433f_9897_4f5ea47f5fe7.slice/crio-fadbc6bbd06d13ed24668b4998c25f1a27f8fb894f154be240ece79ee5c89f64 WatchSource:0}: Error finding container fadbc6bbd06d13ed24668b4998c25f1a27f8fb894f154be240ece79ee5c89f64: Status 404 returned error can't find the container with id fadbc6bbd06d13ed24668b4998c25f1a27f8fb894f154be240ece79ee5c89f64 Jul 21 21:48:45.842248 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:45.842164 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8619db3e_9222_4014_829d_7fe3e363d332.slice/crio-117f314952b4551fd29398ed9995290c2790e5b78e0859fa2754cbf5260e3999 WatchSource:0}: Error finding container 117f314952b4551fd29398ed9995290c2790e5b78e0859fa2754cbf5260e3999: Status 404 returned error can't find the container with id 117f314952b4551fd29398ed9995290c2790e5b78e0859fa2754cbf5260e3999 Jul 21 21:48:45.844941 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:48:45.844919 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8c48b5_51f3_4bbc_bfe4_b8b7e562dc7b.slice/crio-39017efaa4186b13844f99c0d302be588158dd7eb631b2c1597d605044189ea7 WatchSource:0}: Error finding container 39017efaa4186b13844f99c0d302be588158dd7eb631b2c1597d605044189ea7: Status 404 returned error can't find the container with id 39017efaa4186b13844f99c0d302be588158dd7eb631b2c1597d605044189ea7 Jul 21 21:48:46.314133 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.313805 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret\") pod \"global-pull-secret-syncer-fb457\" (UID: \"b3de97b9-626a-4516-9749-94ff62f60273\") " pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:46.314133 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.314094 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs\") pod \"network-metrics-daemon-zxm6p\" (UID: \"052e4590-9613-4a30-b87c-fe56d7a11e9a\") " pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:48:46.314133 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.314126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2qm\" (UniqueName: \"kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm\") pod \"network-check-target-zrk2g\" (UID: \"1b7dcb72-e10b-4f88-be68-79d8b3b97f87\") " pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:48:46.314444 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:46.314294 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:48:46.314444 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:46.314318 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:48:46.314444 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:46.314333 2580 projected.go:196] Error preparing data for projected volume kube-api-access-bz2qm for pod openshift-network-diagnostics/network-check-target-zrk2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:46.314444 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:46.314432 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm podName:1b7dcb72-e10b-4f88-be68-79d8b3b97f87 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:48.314386869 +0000 UTC m=+6.077621129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bz2qm" (UniqueName: "kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm") pod "network-check-target-zrk2g" (UID: "1b7dcb72-e10b-4f88-be68-79d8b3b97f87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:46.314920 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:46.314896 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:46.314998 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:46.314956 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs podName:052e4590-9613-4a30-b87c-fe56d7a11e9a nodeName:}" failed. No retries permitted until 2026-07-21 21:48:48.314939474 +0000 UTC m=+6.078173718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs") pod "network-metrics-daemon-zxm6p" (UID: "052e4590-9613-4a30-b87c-fe56d7a11e9a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:46.315074 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:46.315022 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:46.315074 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:46.315056 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret podName:b3de97b9-626a-4516-9749-94ff62f60273 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:48.315044405 +0000 UTC m=+6.078278647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret") pod "global-pull-secret-syncer-fb457" (UID: "b3de97b9-626a-4516-9749-94ff62f60273") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:46.624885 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.624788 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 21:43:45 +0000 UTC" deadline="2028-02-17 16:35:09.560542592 +0000 UTC" Jul 21 21:48:46.624885 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.624833 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13818h46m22.935715194s" Jul 21 21:48:46.773363 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.773331 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:48:46.773633 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:46.773486 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:48:46.773946 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.773914 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:48:46.774050 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:46.774016 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:48:46.808163 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.808102 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal" event={"ID":"9ba3d3cdb4f4f23180aa70b807d11a56","Type":"ContainerStarted","Data":"dea544998afee586f33bc7b9113328aafd830ac74a4665fb8af1c3d981d469ae"} Jul 21 21:48:46.822353 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.822315 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fk4dn" event={"ID":"5daa5430-2084-48d2-a008-6305a894ea65","Type":"ContainerStarted","Data":"a6177c22453df0da9ae6dfebfe284da192d8a06719a28ff119468f92578cb981"} Jul 21 21:48:46.826490 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.826415 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal" podStartSLOduration=2.826396919 podStartE2EDuration="2.826396919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:48:46.826019761 +0000 UTC m=+4.589254029" watchObservedRunningTime="2026-07-21 21:48:46.826396919 +0000 UTC m=+4.589631185" Jul 21 21:48:46.830001 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.829938 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mxlb5" event={"ID":"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96","Type":"ContainerStarted","Data":"1de1b5e2c7de7bf09381b93e43f919125a62607a60aea95a364827bdae82ed7d"} Jul 21 21:48:46.833389 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.833322 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" event={"ID":"2989142e-cd19-4991-85d2-6c75e18ce1ae","Type":"ContainerStarted","Data":"253bab74830f1db291d1241a82471143b5e0eebedbadeeeb76b2f79ad1baae16"} Jul 21 21:48:46.845358 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.845328 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wjhtb" event={"ID":"0c0ed0aa-b6e4-4b63-8ab3-378c8e1024aa","Type":"ContainerStarted","Data":"4a0cefb7c47084ccbf1e0aed20c65c68470f9744a4de28cd23b0f4184a6f350d"} Jul 21 21:48:46.851217 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.851142 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xq9jl" event={"ID":"7935139c-997c-4f11-83b9-cca9a0053e65","Type":"ContainerStarted","Data":"8031d20ed25796194cd610b236cf6f83a3b33e8cfc372325f7a7deb5234e7fdb"} Jul 21 21:48:46.858310 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.857587 2580 generic.go:356] "Generic (PLEG): container finished" podID="485e76d26b45188751c34edd66993253" containerID="206652a78cf6020313f508ec44c3d3ecd986fc1b56594724a5848b526559ddd6" exitCode=0 Jul 21 21:48:46.858310 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.857670 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal" event={"ID":"485e76d26b45188751c34edd66993253","Type":"ContainerDied","Data":"206652a78cf6020313f508ec44c3d3ecd986fc1b56594724a5848b526559ddd6"} Jul 21 21:48:46.873462 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.872929 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jdcql" event={"ID":"d6c5658d-ed28-4bef-ada7-ab4746a7a346","Type":"ContainerStarted","Data":"4bbe244681fb1fd2e41532de957fcabc26b9176d243049217c2be40cdcbeee29"} Jul 21 21:48:46.884476 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.884342 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" event={"ID":"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b","Type":"ContainerStarted","Data":"39017efaa4186b13844f99c0d302be588158dd7eb631b2c1597d605044189ea7"} Jul 21 21:48:46.896237 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.896138 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" event={"ID":"8619db3e-9222-4014-829d-7fe3e363d332","Type":"ContainerStarted","Data":"117f314952b4551fd29398ed9995290c2790e5b78e0859fa2754cbf5260e3999"} Jul 21 21:48:46.909385 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:46.909329 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6jrl" event={"ID":"08eea2bf-67da-433f-9897-4f5ea47f5fe7","Type":"ContainerStarted","Data":"fadbc6bbd06d13ed24668b4998c25f1a27f8fb894f154be240ece79ee5c89f64"} Jul 21 21:48:47.773247 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:47.773209 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:47.773728 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:47.773374 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:48:47.931905 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:47.931804 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal" event={"ID":"485e76d26b45188751c34edd66993253","Type":"ContainerStarted","Data":"012f67dce2ba24b07cbf5dc71b05f5d27051f351386c0a470a164419952b75ea"} Jul 21 21:48:48.330786 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:48.330744 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret\") pod \"global-pull-secret-syncer-fb457\" (UID: \"b3de97b9-626a-4516-9749-94ff62f60273\") " pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:48.331002 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:48.330809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs\") pod \"network-metrics-daemon-zxm6p\" (UID: \"052e4590-9613-4a30-b87c-fe56d7a11e9a\") " pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:48:48.331002 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:48.330849 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2qm\" (UniqueName: \"kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm\") pod \"network-check-target-zrk2g\" (UID: \"1b7dcb72-e10b-4f88-be68-79d8b3b97f87\") " pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:48:48.331002 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:48.330995 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:48:48.331177 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:48.331014 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:48:48.331177 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:48.331027 2580 projected.go:196] Error preparing data for projected volume kube-api-access-bz2qm for pod openshift-network-diagnostics/network-check-target-zrk2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:48.331177 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:48.331089 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm podName:1b7dcb72-e10b-4f88-be68-79d8b3b97f87 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:52.331069885 +0000 UTC m=+10.094304132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bz2qm" (UniqueName: "kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm") pod "network-check-target-zrk2g" (UID: "1b7dcb72-e10b-4f88-be68-79d8b3b97f87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:48.331546 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:48.331520 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:48.331640 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:48.331585 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret podName:b3de97b9-626a-4516-9749-94ff62f60273 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:52.331567402 +0000 UTC m=+10.094801642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret") pod "global-pull-secret-syncer-fb457" (UID: "b3de97b9-626a-4516-9749-94ff62f60273") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:48.331710 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:48.331646 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:48.331710 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:48.331680 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs podName:052e4590-9613-4a30-b87c-fe56d7a11e9a nodeName:}" failed. No retries permitted until 2026-07-21 21:48:52.331667611 +0000 UTC m=+10.094901856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs") pod "network-metrics-daemon-zxm6p" (UID: "052e4590-9613-4a30-b87c-fe56d7a11e9a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:48.774610 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:48.774528 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:48:48.774610 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:48.774577 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:48:48.775159 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:48.774677 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:48:48.775159 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:48.774821 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:48:49.773167 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:49.773128 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:49.773345 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:49.773286 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:48:50.774181 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:50.774114 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:48:50.774181 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:50.774172 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:48:50.774757 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:50.774307 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:48:50.776591 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:50.776555 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:48:51.773832 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:51.773261 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:51.773832 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:51.773422 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:48:52.367434 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:52.367396 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret\") pod \"global-pull-secret-syncer-fb457\" (UID: \"b3de97b9-626a-4516-9749-94ff62f60273\") " pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:52.367900 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:52.367456 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs\") pod \"network-metrics-daemon-zxm6p\" (UID: \"052e4590-9613-4a30-b87c-fe56d7a11e9a\") " pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:48:52.367900 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:52.367491 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2qm\" (UniqueName: \"kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm\") pod \"network-check-target-zrk2g\" (UID: \"1b7dcb72-e10b-4f88-be68-79d8b3b97f87\") " pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:48:52.367900 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:52.367662 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:48:52.367900 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:52.367682 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:48:52.367900 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:52.367695 2580 projected.go:196] Error preparing data for projected volume kube-api-access-bz2qm for pod openshift-network-diagnostics/network-check-target-zrk2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:52.367900 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:52.367750 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm podName:1b7dcb72-e10b-4f88-be68-79d8b3b97f87 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:00.367732191 +0000 UTC m=+18.130966438 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bz2qm" (UniqueName: "kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm") pod "network-check-target-zrk2g" (UID: "1b7dcb72-e10b-4f88-be68-79d8b3b97f87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:52.368209 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:52.368131 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:52.368209 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:52.368176 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret podName:b3de97b9-626a-4516-9749-94ff62f60273 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:00.368160972 +0000 UTC m=+18.131395215 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret") pod "global-pull-secret-syncer-fb457" (UID: "b3de97b9-626a-4516-9749-94ff62f60273") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:52.368310 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:52.368235 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:52.368310 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:52.368268 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs podName:052e4590-9613-4a30-b87c-fe56d7a11e9a nodeName:}" failed. No retries permitted until 2026-07-21 21:49:00.368256262 +0000 UTC m=+18.131490502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs") pod "network-metrics-daemon-zxm6p" (UID: "052e4590-9613-4a30-b87c-fe56d7a11e9a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:52.774533 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:52.774451 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:48:52.774685 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:52.774563 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:48:52.775060 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:52.775029 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:48:52.775187 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:52.775153 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:48:53.773885 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:53.773421 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:53.773885 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:53.773548 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:48:54.773207 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:54.773173 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:48:54.773364 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:54.773273 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:48:54.773364 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:54.773295 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:48:54.773482 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:54.773410 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:48:55.773403 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:55.772932 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:55.773403 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:55.773064 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:48:56.773643 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:56.773607 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:48:56.774040 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:56.773735 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:48:56.774040 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:56.773825 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:48:56.774040 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:56.773922 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:48:57.773232 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:57.773197 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:57.773427 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:57.773311 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:48:58.773252 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:58.773216 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:48:58.773718 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:58.773339 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:48:58.773718 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:58.773397 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:48:58.773718 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:58.773536 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:48:59.773860 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:48:59.773828 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:48:59.774423 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:48:59.773924 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:00.432356 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:00.432317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret\") pod \"global-pull-secret-syncer-fb457\" (UID: \"b3de97b9-626a-4516-9749-94ff62f60273\") " pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:00.432534 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:00.432390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs\") pod \"network-metrics-daemon-zxm6p\" (UID: \"052e4590-9613-4a30-b87c-fe56d7a11e9a\") " pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:00.432534 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:00.432425 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2qm\" (UniqueName: \"kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm\") pod \"network-check-target-zrk2g\" (UID: \"1b7dcb72-e10b-4f88-be68-79d8b3b97f87\") " pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:00.432534 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:00.432485 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:49:00.432534 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:00.432494 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:49:00.432750 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:00.432549 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:49:00.432750 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:00.432567 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:49:00.432750 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:00.432580 2580 projected.go:196] Error preparing data for projected volume kube-api-access-bz2qm for pod openshift-network-diagnostics/network-check-target-zrk2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:49:00.432750 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:00.432555 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs podName:052e4590-9613-4a30-b87c-fe56d7a11e9a nodeName:}" failed. No retries permitted until 2026-07-21 21:49:16.432538929 +0000 UTC m=+34.195773168 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs") pod "network-metrics-daemon-zxm6p" (UID: "052e4590-9613-4a30-b87c-fe56d7a11e9a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:49:00.432750 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:00.432643 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret podName:b3de97b9-626a-4516-9749-94ff62f60273 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:16.432626613 +0000 UTC m=+34.195860857 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret") pod "global-pull-secret-syncer-fb457" (UID: "b3de97b9-626a-4516-9749-94ff62f60273") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:49:00.432750 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:00.432665 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm podName:1b7dcb72-e10b-4f88-be68-79d8b3b97f87 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:16.432654564 +0000 UTC m=+34.195888808 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bz2qm" (UniqueName: "kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm") pod "network-check-target-zrk2g" (UID: "1b7dcb72-e10b-4f88-be68-79d8b3b97f87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:49:00.773016 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:00.772933 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:00.773164 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:00.773047 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:00.773316 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:00.773295 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:00.773525 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:00.773465 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:01.773649 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:01.773572 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:01.774027 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:01.773699 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:02.774028 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:02.773990 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:02.774472 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:02.774059 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:02.774472 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:02.774139 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:02.774472 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:02.774195 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:03.773684 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:03.773644 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:03.773838 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:03.773759 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:04.773875 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:04.773843 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:04.774451 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:04.773956 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:04.774451 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:04.774015 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:04.774451 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:04.774151 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:05.773814 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:05.773513 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:05.773929 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:05.773897 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:05.982525 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:05.982497 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mxlb5" event={"ID":"6a67a0b6-b146-4679-8ad7-9d1c7fdd7d96","Type":"ContainerStarted","Data":"2fd1796bcc4fee876f33fb0594bb46b453c68e34ccca563acd2a779669096c04"} Jul 21 21:49:05.983879 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:05.983858 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" event={"ID":"2989142e-cd19-4991-85d2-6c75e18ce1ae","Type":"ContainerStarted","Data":"c849865ee4ba438eaf7656beaa29ecb215e3f606f2ccb5d87705761be42381bb"} Jul 21 21:49:05.985194 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:05.985162 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wjhtb" event={"ID":"0c0ed0aa-b6e4-4b63-8ab3-378c8e1024aa","Type":"ContainerStarted","Data":"aca8eda94ece729841ba2d8c43c85e98d4e73af9740bbb6f4ca925c256e39416"} Jul 21 21:49:05.986517 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:05.986492 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jdcql" event={"ID":"d6c5658d-ed28-4bef-ada7-ab4746a7a346","Type":"ContainerStarted","Data":"a5ee122d5bb84096e8b57c728b65f5686932fd31bb2382000a25ebe7e9d0af0e"} Jul 21 21:49:05.987811 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:05.987787 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" event={"ID":"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b","Type":"ContainerStarted","Data":"93881b1728d7f6dc12315996f305bb0af01e28df7770157e079a193762cdd483"} Jul 21 21:49:05.990229 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:05.990211 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" event={"ID":"8619db3e-9222-4014-829d-7fe3e363d332","Type":"ContainerStarted","Data":"b09768133e26dcd8cfc6ebd88d0b55e5ecc0ad65fb5ce5927d7e79418380955b"} Jul 21 21:49:05.990319 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:05.990277 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" event={"ID":"8619db3e-9222-4014-829d-7fe3e363d332","Type":"ContainerStarted","Data":"b0dc2115a5a9e3af4f0728305b7f98bc2463de1d22cae4e8f8a8925bb4ecf60e"} Jul 21 21:49:05.990385 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:05.990326 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" event={"ID":"8619db3e-9222-4014-829d-7fe3e363d332","Type":"ContainerStarted","Data":"b85632b841fd7df63717dc990ab7bd9cc60b6963f45234cfaaf882a53a8de31d"} Jul 21 21:49:05.990435 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:05.990391 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" event={"ID":"8619db3e-9222-4014-829d-7fe3e363d332","Type":"ContainerStarted","Data":"4647976d8e2f357d9dfde6559c900a90788d83b505b3e60e64dd212a08b98511"} Jul 21 21:49:05.990475 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:05.990439 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" event={"ID":"8619db3e-9222-4014-829d-7fe3e363d332","Type":"ContainerStarted","Data":"41c3059021b101538355c651eac12f20cb236bb206d82f79102980830f5b169b"} Jul 21 21:49:05.991641 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:05.991619 2580 generic.go:356] "Generic (PLEG): container finished" podID="08eea2bf-67da-433f-9897-4f5ea47f5fe7" containerID="dd0c7c79c7910f5d9aae4a8f8a97584f865a6be5e7cafdac611a8daa3c246ea0" exitCode=0 Jul 21 21:49:05.991741 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:05.991684 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6jrl" event={"ID":"08eea2bf-67da-433f-9897-4f5ea47f5fe7","Type":"ContainerDied","Data":"dd0c7c79c7910f5d9aae4a8f8a97584f865a6be5e7cafdac611a8daa3c246ea0"} Jul 21 21:49:05.992977 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:05.992879 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fk4dn" event={"ID":"5daa5430-2084-48d2-a008-6305a894ea65","Type":"ContainerStarted","Data":"1c98041b138d044efe12376bde2fb95948fae702479ca423456d90bdf31ad9f4"} Jul 21 21:49:06.004473 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:06.004437 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal" podStartSLOduration=22.004422729 podStartE2EDuration="22.004422729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:48:47.952641308 +0000 UTC m=+5.715875575" watchObservedRunningTime="2026-07-21 21:49:06.004422729 +0000 UTC m=+23.767656991" Jul 21 21:49:06.020760 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:06.020725 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mxlb5" podStartSLOduration=3.510264711 podStartE2EDuration="23.020712088s" totalImagesPullingTime="19.510447377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:06.00368564 +0000 UTC m=+23.766919901" watchObservedRunningTime="2026-07-21 21:49:06.020712088 +0000 UTC m=+23.783946356" Jul 21 21:49:06.021061 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:06.021037 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wjhtb" podStartSLOduration=7.408059483 podStartE2EDuration="23.021031886s" totalImagesPullingTime="15.612972403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:06.020549576 +0000 UTC m=+23.783783838" watchObservedRunningTime="2026-07-21 21:49:06.021031886 +0000 UTC m=+23.784266180" Jul 21 21:49:06.086025 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:06.085941 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5dgn7" podStartSLOduration=3.629605116 podStartE2EDuration="23.085926377s" totalImagesPullingTime="19.456321261s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:06.066701893 +0000 UTC m=+23.829936153" watchObservedRunningTime="2026-07-21 21:49:06.085926377 +0000 UTC m=+23.849161053" Jul 21 21:49:06.086154 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:06.086032 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fk4dn" podStartSLOduration=3.654800158 podStartE2EDuration="23.086028738s" totalImagesPullingTime="19.43122858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:06.085613512 +0000 UTC m=+23.848847772" watchObservedRunningTime="2026-07-21 21:49:06.086028738 +0000 UTC m=+23.849263001" Jul 21 21:49:06.101346 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:06.101307 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jdcql" podStartSLOduration=7.4977247 podStartE2EDuration="23.101291949s" totalImagesPullingTime="15.603567249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:06.101233851 +0000 UTC m=+23.864468115" watchObservedRunningTime="2026-07-21 21:49:06.101291949 +0000 UTC m=+23.864526210" Jul 21 21:49:06.773115 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:06.773088 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:06.773298 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:06.773088 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:06.773298 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:06.773215 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:06.773412 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:06.773296 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:06.988771 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:06.988745 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 21:49:06.999893 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:06.999868 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" event={"ID":"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b","Type":"ContainerStarted","Data":"6624e61d8c2b7e8b0f53a4f67a6964f8ea239d374c34a6f5895775e765c17f4e"} Jul 21 21:49:07.003211 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:07.003190 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" event={"ID":"8619db3e-9222-4014-829d-7fe3e363d332","Type":"ContainerStarted","Data":"b24592eae46c90f1a7bc1f54aec9fd7d25b4da7d266087e5c6b2e4685af2561d"} Jul 21 21:49:07.004760 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:07.004736 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xq9jl" event={"ID":"7935139c-997c-4f11-83b9-cca9a0053e65","Type":"ContainerStarted","Data":"b8473af79084a2deb27d280a0ebbd70ed92327cd9e47e6c60f7b857732920bbf"} Jul 21 21:49:07.021225 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:07.021186 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xq9jl" podStartSLOduration=4.573971768 podStartE2EDuration="24.021171293s" totalImagesPullingTime="19.447199525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:07.020957231 +0000 UTC m=+24.784191494" watchObservedRunningTime="2026-07-21 21:49:07.021171293 +0000 UTC m=+24.784405556" Jul 21 21:49:07.665522 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:07.665410 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T21:49:06.988768428Z","UUID":"6e15a0ec-a20d-44f4-bd52-80bcdb0fa093","Handler":null,"Name":"","Endpoint":""} Jul 21 21:49:07.667152 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:07.667126 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 21:49:07.667270 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:07.667173 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 21:49:07.773610 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:07.773573 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:07.773796 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:07.773707 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:08.618425 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:08.618396 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wjhtb" Jul 21 21:49:08.619167 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:08.619148 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wjhtb" Jul 21 21:49:08.773698 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:08.773672 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:08.773875 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:08.773673 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:08.773875 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:08.773772 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:08.773974 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:08.773878 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:09.012596 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:09.012562 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" event={"ID":"4d8c48b5-51f3-4bbc-bfe4-b8b7e562dc7b","Type":"ContainerStarted","Data":"feb3df37d8168576abbc4d6acf045f7e73cbb01156d044169079f1dfb6047c1b"} Jul 21 21:49:09.017039 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:09.017013 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" event={"ID":"8619db3e-9222-4014-829d-7fe3e363d332","Type":"ContainerStarted","Data":"4c4dd87d3f29c243a3c9067f80d1643b43e94c80e08a27d87e8ab62e83b22c5a"} Jul 21 21:49:09.017179 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:09.017102 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wjhtb" Jul 21 21:49:09.017556 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:09.017540 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wjhtb" Jul 21 21:49:09.032342 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:09.032304 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-przmk" podStartSLOduration=4.01523224 podStartE2EDuration="26.032292212s" totalImagesPullingTime="22.017059972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:09.032145261 +0000 UTC m=+26.795379526" watchObservedRunningTime="2026-07-21 21:49:09.032292212 +0000 UTC m=+26.795526474" Jul 21 21:49:09.772911 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:09.772886 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:09.773364 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:09.773015 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:10.773596 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:10.773422 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:10.774238 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:10.773459 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:10.774238 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:10.773683 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:10.774238 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:10.773767 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:11.024045 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:11.024010 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" event={"ID":"8619db3e-9222-4014-829d-7fe3e363d332","Type":"ContainerStarted","Data":"80a40f58cfe2ff173081d5087a7b45f1131b0306040a2101358e0aab131ea533"} Jul 21 21:49:11.024406 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:11.024216 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:49:11.024480 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:11.024423 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:49:11.024527 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:11.024498 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:49:11.026124 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:11.026092 2580 generic.go:356] "Generic (PLEG): container finished" podID="08eea2bf-67da-433f-9897-4f5ea47f5fe7" containerID="1f6f7297f1e97f51c39c8f443bfd575ea9af5df4c647902a88e0e2480a152c90" exitCode=0 Jul 21 21:49:11.026253 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:11.026127 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6jrl" event={"ID":"08eea2bf-67da-433f-9897-4f5ea47f5fe7","Type":"ContainerDied","Data":"1f6f7297f1e97f51c39c8f443bfd575ea9af5df4c647902a88e0e2480a152c90"} Jul 21 21:49:11.042877 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:11.042853 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:49:11.043069 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:11.042941 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:49:11.053009 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:11.052974 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" podStartSLOduration=8.54464917 podStartE2EDuration="28.052963336s" totalImagesPullingTime="19.508314166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:11.051459233 +0000 UTC m=+28.814693493" watchObservedRunningTime="2026-07-21 21:49:11.052963336 +0000 UTC m=+28.816197596" Jul 21 21:49:11.476090 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:11.476011 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fk4dn_5daa5430-2084-48d2-a008-6305a894ea65/dns-node-resolver/0.log" Jul 21 21:49:11.774044 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:11.773568 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:11.774044 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:11.773697 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:12.652771 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:12.652557 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jdcql_d6c5658d-ed28-4bef-ada7-ab4746a7a346/node-ca/0.log" Jul 21 21:49:12.681284 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:12.681251 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fb457"] Jul 21 21:49:12.681457 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:12.681366 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:12.681529 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:12.681491 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:12.682190 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:12.682164 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zxm6p"] Jul 21 21:49:12.682315 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:12.682273 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:12.682445 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:12.682419 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:12.682919 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:12.682900 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zrk2g"] Jul 21 21:49:12.682990 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:12.682983 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:12.683063 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:12.683046 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:13.032526 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:13.032496 2580 generic.go:356] "Generic (PLEG): container finished" podID="08eea2bf-67da-433f-9897-4f5ea47f5fe7" containerID="469fb274fe2b1ff49d2e012a9f020fa84ecc8b7081a7903cdbf13b85ab19bbf4" exitCode=0 Jul 21 21:49:13.032907 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:13.032581 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6jrl" event={"ID":"08eea2bf-67da-433f-9897-4f5ea47f5fe7","Type":"ContainerDied","Data":"469fb274fe2b1ff49d2e012a9f020fa84ecc8b7081a7903cdbf13b85ab19bbf4"} Jul 21 21:49:13.773177 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:13.773148 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:13.773334 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:13.773249 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:14.773221 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:14.773197 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:14.773586 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:14.773196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:14.773586 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:14.773310 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:14.773586 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:14.773398 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:15.038970 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:15.038900 2580 generic.go:356] "Generic (PLEG): container finished" podID="08eea2bf-67da-433f-9897-4f5ea47f5fe7" containerID="cb4d03c55d99c58fab64ee4047e199d61ab16b5fcc2e733ba2852e62bf258cd9" exitCode=0 Jul 21 21:49:15.038970 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:15.038949 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6jrl" event={"ID":"08eea2bf-67da-433f-9897-4f5ea47f5fe7","Type":"ContainerDied","Data":"cb4d03c55d99c58fab64ee4047e199d61ab16b5fcc2e733ba2852e62bf258cd9"} Jul 21 21:49:15.773829 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:15.773798 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:15.774193 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:15.773928 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:16.454938 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:16.454903 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2qm\" (UniqueName: \"kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm\") pod \"network-check-target-zrk2g\" (UID: \"1b7dcb72-e10b-4f88-be68-79d8b3b97f87\") " pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:16.455105 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:16.454969 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret\") pod \"global-pull-secret-syncer-fb457\" (UID: \"b3de97b9-626a-4516-9749-94ff62f60273\") " pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:16.455105 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:16.454999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs\") pod \"network-metrics-daemon-zxm6p\" (UID: \"052e4590-9613-4a30-b87c-fe56d7a11e9a\") " pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:16.455105 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:16.455076 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:49:16.455105 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:16.455100 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:49:16.455105 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:16.455101 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:49:16.455525 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:16.455103 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:49:16.455525 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:16.455113 2580 projected.go:196] Error preparing data for projected volume kube-api-access-bz2qm for pod openshift-network-diagnostics/network-check-target-zrk2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:49:16.455525 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:16.455165 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret podName:b3de97b9-626a-4516-9749-94ff62f60273 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:48.455144822 +0000 UTC m=+66.218379062 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret") pod "global-pull-secret-syncer-fb457" (UID: "b3de97b9-626a-4516-9749-94ff62f60273") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:49:16.455525 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:16.455181 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm podName:1b7dcb72-e10b-4f88-be68-79d8b3b97f87 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:48.455173263 +0000 UTC m=+66.218407512 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bz2qm" (UniqueName: "kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm") pod "network-check-target-zrk2g" (UID: "1b7dcb72-e10b-4f88-be68-79d8b3b97f87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:49:16.455525 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:16.455195 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs podName:052e4590-9613-4a30-b87c-fe56d7a11e9a nodeName:}" failed. No retries permitted until 2026-07-21 21:49:48.455187537 +0000 UTC m=+66.218421776 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs") pod "network-metrics-daemon-zxm6p" (UID: "052e4590-9613-4a30-b87c-fe56d7a11e9a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:49:16.777520 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:16.777489 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:16.777888 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:16.777615 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:16.777888 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:16.777634 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:16.777888 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:16.777724 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:17.773344 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:17.773315 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:17.773540 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:17.773448 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:18.776482 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:18.776219 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:18.776888 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:18.776279 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:18.776888 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:18.776590 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:18.776888 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:18.776653 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:19.773542 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:19.773508 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:19.773696 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:19.773633 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:20.053563 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:20.053480 2580 generic.go:356] "Generic (PLEG): container finished" podID="08eea2bf-67da-433f-9897-4f5ea47f5fe7" containerID="57be2e5a4dc86debdad02112a92f2c3a577ae0b9296b0726a6b9a222cfb56eb0" exitCode=0 Jul 21 21:49:20.053563 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:20.053543 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6jrl" event={"ID":"08eea2bf-67da-433f-9897-4f5ea47f5fe7","Type":"ContainerDied","Data":"57be2e5a4dc86debdad02112a92f2c3a577ae0b9296b0726a6b9a222cfb56eb0"} Jul 21 21:49:20.773233 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:20.773206 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:20.773391 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:20.773261 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:20.773391 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:20.773329 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:20.773469 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:20.773424 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:21.060757 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:21.059392 2580 generic.go:356] "Generic (PLEG): container finished" podID="08eea2bf-67da-433f-9897-4f5ea47f5fe7" containerID="8be26961a94b5e5c7b0119e05c5eed0d43d9f7976235f3c08281267bc9cd7548" exitCode=0 Jul 21 21:49:21.060757 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:21.059469 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6jrl" event={"ID":"08eea2bf-67da-433f-9897-4f5ea47f5fe7","Type":"ContainerDied","Data":"8be26961a94b5e5c7b0119e05c5eed0d43d9f7976235f3c08281267bc9cd7548"} Jul 21 21:49:21.773414 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:21.773367 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:21.773563 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:21.773483 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:22.065007 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:22.064978 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6jrl" event={"ID":"08eea2bf-67da-433f-9897-4f5ea47f5fe7","Type":"ContainerStarted","Data":"40723a8175a4b4062397e434578edfb96b16159689ba8617176daed545920051"} Jul 21 21:49:22.089921 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:22.089884 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t6jrl" podStartSLOduration=9.142700519 podStartE2EDuration="39.089869716s" totalImagesPullingTime="29.947169197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:22.089611624 +0000 UTC m=+39.852845886" watchObservedRunningTime="2026-07-21 21:49:22.089869716 +0000 UTC m=+39.853103977" Jul 21 21:49:22.773651 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:22.773619 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:22.773850 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:22.773711 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:22.773850 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:22.773797 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:22.773932 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:22.773868 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:23.773688 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:23.773657 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:23.774135 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:23.773759 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:24.773300 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:24.773275 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:24.773485 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:24.773281 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:24.773485 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:24.773383 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:24.773485 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:24.773455 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:25.772973 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:25.772944 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:25.773302 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:25.773032 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:26.773149 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:26.773117 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:26.773149 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:26.773140 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:26.773710 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:26.773219 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:26.773710 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:26.773344 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:27.773656 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:27.773619 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:27.774012 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:27.773728 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:28.773695 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:28.773660 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:28.774132 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:28.773765 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:28.774132 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:28.773821 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:28.774132 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:28.773913 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:29.772992 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:29.772959 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:29.773158 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:29.773060 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:30.772879 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:30.772850 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:30.773229 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:30.772854 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:30.773229 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:30.772952 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:30.773229 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:30.773023 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:31.773151 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:31.773120 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:31.773579 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:31.773233 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:32.773790 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:32.773761 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:32.774152 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:32.773861 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:32.774152 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:32.773969 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:32.774152 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:32.774052 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:33.772799 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:33.772772 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:33.772954 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:33.772859 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:34.773606 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:34.773575 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:34.773986 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:34.773711 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:34.773986 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:34.773761 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:34.773986 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:34.773838 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:35.772840 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:35.772814 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:35.773070 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:35.772910 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:36.773229 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:36.773200 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:36.773642 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:36.773241 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:36.773642 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:36.773292 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:36.773642 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:36.773430 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:37.773760 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:37.773727 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:37.774174 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:37.773847 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fb457" podUID="b3de97b9-626a-4516-9749-94ff62f60273" Jul 21 21:49:38.773122 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:38.773086 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:38.773288 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:38.773207 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zrk2g" podUID="1b7dcb72-e10b-4f88-be68-79d8b3b97f87" Jul 21 21:49:38.773359 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:38.773289 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:38.773456 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:38.773435 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxm6p" podUID="052e4590-9613-4a30-b87c-fe56d7a11e9a" Jul 21 21:49:39.041876 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.041802 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeReady" Jul 21 21:49:39.042221 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.041904 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 21:49:39.092217 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.092188 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5l4xt"] Jul 21 21:49:39.094001 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.093986 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5l4xt" Jul 21 21:49:39.096427 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.096404 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 21:49:39.096531 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.096491 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pp5jq\"" Jul 21 21:49:39.096531 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.096491 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 21:49:39.099110 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.099092 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pdv8n"] Jul 21 21:49:39.101266 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.101250 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pdv8n" Jul 21 21:49:39.103594 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.103576 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 21:49:39.103763 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.103743 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 21:49:39.103865 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.103844 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 21:49:39.104292 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.104275 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 21:49:39.104706 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.104689 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6tqm4\"" Jul 21 21:49:39.104973 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.104952 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5l4xt"] Jul 21 21:49:39.114818 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.114796 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pdv8n"] Jul 21 21:49:39.127451 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.127433 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a67aeca-9eb3-48be-89b6-27696cefec9d-tmp-dir\") pod \"dns-default-5l4xt\" (UID: \"8a67aeca-9eb3-48be-89b6-27696cefec9d\") " pod="openshift-dns/dns-default-5l4xt" Jul 21 21:49:39.127545 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.127458 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r6qf\" (UniqueName: \"kubernetes.io/projected/6eb0fb30-a50c-4913-b8e0-907e9fc18c7a-kube-api-access-2r6qf\") pod \"insights-runtime-extractor-pdv8n\" (UID: \"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a\") " pod="openshift-insights/insights-runtime-extractor-pdv8n" Jul 21 21:49:39.127545 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.127502 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6eb0fb30-a50c-4913-b8e0-907e9fc18c7a-crio-socket\") pod \"insights-runtime-extractor-pdv8n\" (UID: \"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a\") " pod="openshift-insights/insights-runtime-extractor-pdv8n" Jul 21 21:49:39.127545 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.127522 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6eb0fb30-a50c-4913-b8e0-907e9fc18c7a-data-volume\") pod \"insights-runtime-extractor-pdv8n\" (UID: \"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a\") " pod="openshift-insights/insights-runtime-extractor-pdv8n" Jul 21 21:49:39.127643 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.127562 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eb0fb30-a50c-4913-b8e0-907e9fc18c7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pdv8n\" (UID: \"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a\") " pod="openshift-insights/insights-runtime-extractor-pdv8n" Jul 21 21:49:39.127643 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.127597 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rxjx\" (UniqueName: \"kubernetes.io/projected/8a67aeca-9eb3-48be-89b6-27696cefec9d-kube-api-access-4rxjx\") pod \"dns-default-5l4xt\" (UID: \"8a67aeca-9eb3-48be-89b6-27696cefec9d\") " pod="openshift-dns/dns-default-5l4xt" Jul 21 21:49:39.127643 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.127613 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eb0fb30-a50c-4913-b8e0-907e9fc18c7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pdv8n\" (UID: \"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a\") " pod="openshift-insights/insights-runtime-extractor-pdv8n" Jul 21 21:49:39.127643 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.127633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8a67aeca-9eb3-48be-89b6-27696cefec9d-config-volume\") pod \"dns-default-5l4xt\" (UID: \"8a67aeca-9eb3-48be-89b6-27696cefec9d\") " pod="openshift-dns/dns-default-5l4xt" Jul 21 21:49:39.127775 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.127653 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a67aeca-9eb3-48be-89b6-27696cefec9d-metrics-tls\") pod \"dns-default-5l4xt\" (UID: \"8a67aeca-9eb3-48be-89b6-27696cefec9d\") " pod="openshift-dns/dns-default-5l4xt" Jul 21 21:49:39.189939 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.189912 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hdhhq"] Jul 21 21:49:39.191630 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.191613 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hdhhq" Jul 21 21:49:39.194264 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.194230 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 21:49:39.194357 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.194276 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 21:49:39.194357 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.194349 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7wwbw\"" Jul 21 21:49:39.194492 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.194447 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 21:49:39.200892 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.200869 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hdhhq"] Jul 21 21:49:39.228684 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.228667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8a67aeca-9eb3-48be-89b6-27696cefec9d-config-volume\") pod \"dns-default-5l4xt\" (UID: \"8a67aeca-9eb3-48be-89b6-27696cefec9d\") " pod="openshift-dns/dns-default-5l4xt" Jul 21 21:49:39.228782 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.228695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a67aeca-9eb3-48be-89b6-27696cefec9d-metrics-tls\") pod \"dns-default-5l4xt\" (UID: \"8a67aeca-9eb3-48be-89b6-27696cefec9d\") " pod="openshift-dns/dns-default-5l4xt" Jul 21 21:49:39.228782 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.228732 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a67aeca-9eb3-48be-89b6-27696cefec9d-tmp-dir\") pod \"dns-default-5l4xt\" (UID: \"8a67aeca-9eb3-48be-89b6-27696cefec9d\") " pod="openshift-dns/dns-default-5l4xt" Jul 21 21:49:39.228782 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.228746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r6qf\" (UniqueName: \"kubernetes.io/projected/6eb0fb30-a50c-4913-b8e0-907e9fc18c7a-kube-api-access-2r6qf\") pod \"insights-runtime-extractor-pdv8n\" (UID: \"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a\") " pod="openshift-insights/insights-runtime-extractor-pdv8n" Jul 21 21:49:39.228782 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.228778 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/529698f9-3ae6-4d6a-a0dc-839a22bc5119-cert\") pod \"ingress-canary-hdhhq\" (UID: \"529698f9-3ae6-4d6a-a0dc-839a22bc5119\") " pod="openshift-ingress-canary/ingress-canary-hdhhq" Jul 21 21:49:39.228979 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.228799 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6eb0fb30-a50c-4913-b8e0-907e9fc18c7a-crio-socket\") pod \"insights-runtime-extractor-pdv8n\" (UID: \"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a\") " pod="openshift-insights/insights-runtime-extractor-pdv8n" Jul 21 21:49:39.228979 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.228819 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6eb0fb30-a50c-4913-b8e0-907e9fc18c7a-data-volume\") pod \"insights-runtime-extractor-pdv8n\" (UID: \"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a\") " pod="openshift-insights/insights-runtime-extractor-pdv8n" Jul 21 21:49:39.228979 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.228838 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eb0fb30-a50c-4913-b8e0-907e9fc18c7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pdv8n\" (UID: \"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a\") " pod="openshift-insights/insights-runtime-extractor-pdv8n" Jul 21 21:49:39.228979 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.228855 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv8w8\" (UniqueName: \"kubernetes.io/projected/529698f9-3ae6-4d6a-a0dc-839a22bc5119-kube-api-access-bv8w8\") pod \"ingress-canary-hdhhq\" (UID: \"529698f9-3ae6-4d6a-a0dc-839a22bc5119\") " pod="openshift-ingress-canary/ingress-canary-hdhhq" Jul 21 21:49:39.228979 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.228865 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6eb0fb30-a50c-4913-b8e0-907e9fc18c7a-crio-socket\") pod \"insights-runtime-extractor-pdv8n\" (UID: \"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a\") " pod="openshift-insights/insights-runtime-extractor-pdv8n" Jul 21 21:49:39.228979 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.228893 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rxjx\" (UniqueName: \"kubernetes.io/projected/8a67aeca-9eb3-48be-89b6-27696cefec9d-kube-api-access-4rxjx\") pod \"dns-default-5l4xt\" (UID: \"8a67aeca-9eb3-48be-89b6-27696cefec9d\") " pod="openshift-dns/dns-default-5l4xt" Jul 21 21:49:39.228979 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.228918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eb0fb30-a50c-4913-b8e0-907e9fc18c7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pdv8n\" (UID: \"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a\") " pod="openshift-insights/insights-runtime-extractor-pdv8n" Jul 21 21:49:39.229251 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.229115 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a67aeca-9eb3-48be-89b6-27696cefec9d-tmp-dir\") pod \"dns-default-5l4xt\" (UID: \"8a67aeca-9eb3-48be-89b6-27696cefec9d\") " pod="openshift-dns/dns-default-5l4xt" Jul 21 21:49:39.229251 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.229200 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6eb0fb30-a50c-4913-b8e0-907e9fc18c7a-data-volume\") pod \"insights-runtime-extractor-pdv8n\" (UID: \"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a\") " pod="openshift-insights/insights-runtime-extractor-pdv8n" Jul 21 21:49:39.229324 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.229307 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8a67aeca-9eb3-48be-89b6-27696cefec9d-config-volume\") pod \"dns-default-5l4xt\" (UID: \"8a67aeca-9eb3-48be-89b6-27696cefec9d\") " pod="openshift-dns/dns-default-5l4xt" Jul 21 21:49:39.229494 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.229472 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eb0fb30-a50c-4913-b8e0-907e9fc18c7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pdv8n\" (UID: \"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a\") " pod="openshift-insights/insights-runtime-extractor-pdv8n" Jul 21 21:49:39.233003 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.232983 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a67aeca-9eb3-48be-89b6-27696cefec9d-metrics-tls\") pod \"dns-default-5l4xt\" (UID: \"8a67aeca-9eb3-48be-89b6-27696cefec9d\") " pod="openshift-dns/dns-default-5l4xt" Jul 21 21:49:39.233084 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.232983 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eb0fb30-a50c-4913-b8e0-907e9fc18c7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pdv8n\" (UID: \"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a\") " pod="openshift-insights/insights-runtime-extractor-pdv8n" Jul 21 21:49:39.237195 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.237174 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rxjx\" (UniqueName: \"kubernetes.io/projected/8a67aeca-9eb3-48be-89b6-27696cefec9d-kube-api-access-4rxjx\") pod \"dns-default-5l4xt\" (UID: \"8a67aeca-9eb3-48be-89b6-27696cefec9d\") " pod="openshift-dns/dns-default-5l4xt" Jul 21 21:49:39.237429 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.237408 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r6qf\" (UniqueName: \"kubernetes.io/projected/6eb0fb30-a50c-4913-b8e0-907e9fc18c7a-kube-api-access-2r6qf\") pod \"insights-runtime-extractor-pdv8n\" (UID: \"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a\") " pod="openshift-insights/insights-runtime-extractor-pdv8n" Jul 21 21:49:39.329401 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.329335 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/529698f9-3ae6-4d6a-a0dc-839a22bc5119-cert\") pod \"ingress-canary-hdhhq\" (UID: \"529698f9-3ae6-4d6a-a0dc-839a22bc5119\") " pod="openshift-ingress-canary/ingress-canary-hdhhq" Jul 21 21:49:39.329401 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.329363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv8w8\" (UniqueName: \"kubernetes.io/projected/529698f9-3ae6-4d6a-a0dc-839a22bc5119-kube-api-access-bv8w8\") pod \"ingress-canary-hdhhq\" (UID: \"529698f9-3ae6-4d6a-a0dc-839a22bc5119\") " pod="openshift-ingress-canary/ingress-canary-hdhhq" Jul 21 21:49:39.331432 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.331417 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/529698f9-3ae6-4d6a-a0dc-839a22bc5119-cert\") pod \"ingress-canary-hdhhq\" (UID: \"529698f9-3ae6-4d6a-a0dc-839a22bc5119\") " pod="openshift-ingress-canary/ingress-canary-hdhhq" Jul 21 21:49:39.337789 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.337763 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv8w8\" (UniqueName: \"kubernetes.io/projected/529698f9-3ae6-4d6a-a0dc-839a22bc5119-kube-api-access-bv8w8\") pod \"ingress-canary-hdhhq\" (UID: \"529698f9-3ae6-4d6a-a0dc-839a22bc5119\") " pod="openshift-ingress-canary/ingress-canary-hdhhq" Jul 21 21:49:39.405023 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.405000 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5l4xt" Jul 21 21:49:39.411302 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.411285 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pdv8n" Jul 21 21:49:39.500005 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.499802 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hdhhq" Jul 21 21:49:39.554637 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.554610 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pdv8n"] Jul 21 21:49:39.559358 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:49:39.559325 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb0fb30_a50c_4913_b8e0_907e9fc18c7a.slice/crio-5453a3df97518ead72895e9341f954ecfcedb0d9674851761c65b2c4a158fb8e WatchSource:0}: Error finding container 5453a3df97518ead72895e9341f954ecfcedb0d9674851761c65b2c4a158fb8e: Status 404 returned error can't find the container with id 5453a3df97518ead72895e9341f954ecfcedb0d9674851761c65b2c4a158fb8e Jul 21 21:49:39.568498 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.568474 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5l4xt"] Jul 21 21:49:39.570449 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:49:39.570426 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a67aeca_9eb3_48be_89b6_27696cefec9d.slice/crio-6cfef335206ab18e72da350e05fe516995b05c3261d8ed74486abad01a32785a WatchSource:0}: Error finding container 6cfef335206ab18e72da350e05fe516995b05c3261d8ed74486abad01a32785a: Status 404 returned error can't find the container with id 6cfef335206ab18e72da350e05fe516995b05c3261d8ed74486abad01a32785a Jul 21 21:49:39.623159 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.623133 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hdhhq"] Jul 21 21:49:39.625614 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:49:39.625589 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529698f9_3ae6_4d6a_a0dc_839a22bc5119.slice/crio-5afc3d86aed3c0814b800e7121c142e1e520a65c9f0c0860a59157c5c79a18e3 WatchSource:0}: Error finding container 5afc3d86aed3c0814b800e7121c142e1e520a65c9f0c0860a59157c5c79a18e3: Status 404 returned error can't find the container with id 5afc3d86aed3c0814b800e7121c142e1e520a65c9f0c0860a59157c5c79a18e3 Jul 21 21:49:39.773202 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.773172 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:39.775830 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:39.775810 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 21:49:40.112487 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:40.112452 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hdhhq" event={"ID":"529698f9-3ae6-4d6a-a0dc-839a22bc5119","Type":"ContainerStarted","Data":"5afc3d86aed3c0814b800e7121c142e1e520a65c9f0c0860a59157c5c79a18e3"} Jul 21 21:49:40.114579 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:40.114547 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pdv8n" event={"ID":"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a","Type":"ContainerStarted","Data":"5453a3df97518ead72895e9341f954ecfcedb0d9674851761c65b2c4a158fb8e"} Jul 21 21:49:40.117122 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:40.117050 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5l4xt" event={"ID":"8a67aeca-9eb3-48be-89b6-27696cefec9d","Type":"ContainerStarted","Data":"6cfef335206ab18e72da350e05fe516995b05c3261d8ed74486abad01a32785a"} Jul 21 21:49:40.775044 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:40.773766 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:40.775044 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:40.774997 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:40.777174 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:40.777151 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fd2qt\"" Jul 21 21:49:40.777942 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:40.777918 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 21:49:40.777942 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:40.777972 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 21:49:40.778388 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:40.778353 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 21:49:40.778825 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:40.778669 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n9lxs\"" Jul 21 21:49:43.045766 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:43.045727 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5rqkj" Jul 21 21:49:45.288872 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.288842 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-jhvdw"] Jul 21 21:49:45.310744 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.310721 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-jhvdw"] Jul 21 21:49:45.310886 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.310822 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-jhvdw" Jul 21 21:49:45.313408 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.313360 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 21 21:49:45.313545 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.313448 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4hxgw\"" Jul 21 21:49:45.322794 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.322773 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54f5d79568-5vmjl"] Jul 21 21:49:45.338441 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.338422 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54f5d79568-5vmjl"] Jul 21 21:49:45.338563 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.338523 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.341439 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.341417 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 21 21:49:45.341439 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.341433 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 21:49:45.341606 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.341463 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2fvdp\"" Jul 21 21:49:45.341677 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.341614 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 21 21:49:45.341790 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.341771 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 21 21:49:45.341908 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.341879 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 21:49:45.341952 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.341909 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 21 21:49:45.342177 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.342160 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 21 21:49:45.347122 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.347105 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 21 21:49:45.379980 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.379961 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-service-ca\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.380067 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.379989 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-trusted-ca-bundle\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.380067 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.380060 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-oauth-serving-cert\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.380172 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.380083 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/80b73dfc-bb0f-4fc1-ae01-069daef10f00-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-jhvdw\" (UID: \"80b73dfc-bb0f-4fc1-ae01-069daef10f00\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-jhvdw" Jul 21 21:49:45.380219 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.380162 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c341c9f-8559-4335-8624-82f38b0cdbeb-console-oauth-config\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.380262 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.380244 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-console-config\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.380296 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.380278 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbrsr\" (UniqueName: \"kubernetes.io/projected/6c341c9f-8559-4335-8624-82f38b0cdbeb-kube-api-access-jbrsr\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.380338 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.380300 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c341c9f-8559-4335-8624-82f38b0cdbeb-console-serving-cert\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.481426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.481335 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-oauth-serving-cert\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.481426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.481405 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/80b73dfc-bb0f-4fc1-ae01-069daef10f00-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-jhvdw\" (UID: \"80b73dfc-bb0f-4fc1-ae01-069daef10f00\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-jhvdw" Jul 21 21:49:45.481636 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.481460 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c341c9f-8559-4335-8624-82f38b0cdbeb-console-oauth-config\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.481636 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.481506 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-console-config\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.481636 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.481532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbrsr\" (UniqueName: \"kubernetes.io/projected/6c341c9f-8559-4335-8624-82f38b0cdbeb-kube-api-access-jbrsr\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.481636 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.481562 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c341c9f-8559-4335-8624-82f38b0cdbeb-console-serving-cert\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.481636 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:45.481591 2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 21 21:49:45.481636 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.481607 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-service-ca\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.481636 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.481631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-trusted-ca-bundle\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.481980 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:49:45.481969 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80b73dfc-bb0f-4fc1-ae01-069daef10f00-tls-certificates podName:80b73dfc-bb0f-4fc1-ae01-069daef10f00 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:45.981947153 +0000 UTC m=+63.745181412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/80b73dfc-bb0f-4fc1-ae01-069daef10f00-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-jhvdw" (UID: "80b73dfc-bb0f-4fc1-ae01-069daef10f00") : secret "prometheus-operator-admission-webhook-tls" not found Jul 21 21:49:45.482190 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.482163 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-oauth-serving-cert\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.482315 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.482284 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-console-config\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.482597 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.482579 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-service-ca\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.482831 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.482799 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-trusted-ca-bundle\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.484419 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.484396 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c341c9f-8559-4335-8624-82f38b0cdbeb-console-oauth-config\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.484556 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.484538 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c341c9f-8559-4335-8624-82f38b0cdbeb-console-serving-cert\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.493228 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.493208 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbrsr\" (UniqueName: \"kubernetes.io/projected/6c341c9f-8559-4335-8624-82f38b0cdbeb-kube-api-access-jbrsr\") pod \"console-54f5d79568-5vmjl\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.648863 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.648835 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:45.828596 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.828524 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54f5d79568-5vmjl"] Jul 21 21:49:45.833014 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:49:45.832976 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c341c9f_8559_4335_8624_82f38b0cdbeb.slice/crio-b852a63d4004de97c19c6b6f1c56546da9527108811f2ab0caa2ed3f3bad0d42 WatchSource:0}: Error finding container b852a63d4004de97c19c6b6f1c56546da9527108811f2ab0caa2ed3f3bad0d42: Status 404 returned error can't find the container with id b852a63d4004de97c19c6b6f1c56546da9527108811f2ab0caa2ed3f3bad0d42 Jul 21 21:49:45.984628 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.984589 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/80b73dfc-bb0f-4fc1-ae01-069daef10f00-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-jhvdw\" (UID: \"80b73dfc-bb0f-4fc1-ae01-069daef10f00\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-jhvdw" Jul 21 21:49:45.987177 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:45.987161 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/80b73dfc-bb0f-4fc1-ae01-069daef10f00-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-jhvdw\" (UID: \"80b73dfc-bb0f-4fc1-ae01-069daef10f00\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-jhvdw" Jul 21 21:49:46.137090 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:46.137007 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5l4xt" event={"ID":"8a67aeca-9eb3-48be-89b6-27696cefec9d","Type":"ContainerStarted","Data":"de5073bf7896476c2a77faba31f1f288932f6d3ba309adb245eb595458bd9b7c"} Jul 21 21:49:46.137090 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:46.137075 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5l4xt" event={"ID":"8a67aeca-9eb3-48be-89b6-27696cefec9d","Type":"ContainerStarted","Data":"6b49dbc10787eeab8353de94c830798df4e8c2e6c5aed20ba65089cef87c3d7d"} Jul 21 21:49:46.137269 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:46.137203 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5l4xt" Jul 21 21:49:46.139292 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:46.139228 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f5d79568-5vmjl" event={"ID":"6c341c9f-8559-4335-8624-82f38b0cdbeb","Type":"ContainerStarted","Data":"b852a63d4004de97c19c6b6f1c56546da9527108811f2ab0caa2ed3f3bad0d42"} Jul 21 21:49:46.142128 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:46.142100 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hdhhq" event={"ID":"529698f9-3ae6-4d6a-a0dc-839a22bc5119","Type":"ContainerStarted","Data":"8bae40cc9660a2e7ac275091b62597d46c72da3d6aa6401c52b2f7b031497a4a"} Jul 21 21:49:46.144648 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:46.144625 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pdv8n" event={"ID":"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a","Type":"ContainerStarted","Data":"9f686c6a5cb2725f2330c7413d602f77a77f9fa0da9fff0d128f7828f485bf52"} Jul 21 21:49:46.155595 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:46.155554 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5l4xt" podStartSLOduration=1.505732906 podStartE2EDuration="7.155540943s" totalImagesPullingTime="5.649808037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:46.154777763 +0000 UTC m=+63.918012023" watchObservedRunningTime="2026-07-21 21:49:46.155540943 +0000 UTC m=+63.918775204" Jul 21 21:49:46.171678 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:46.171636 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hdhhq" podStartSLOduration=1.2173355479999999 podStartE2EDuration="7.171618971s" totalImagesPullingTime="5.954283423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:46.170216937 +0000 UTC m=+63.933451211" watchObservedRunningTime="2026-07-21 21:49:46.171618971 +0000 UTC m=+63.934853233" Jul 21 21:49:46.221086 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:46.221061 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-jhvdw" Jul 21 21:49:46.379359 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:46.379327 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-jhvdw"] Jul 21 21:49:46.383346 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:49:46.383323 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b73dfc_bb0f_4fc1_ae01_069daef10f00.slice/crio-fbd5903d67fe5b47527b0e2b004c1109cc8d3f5c1ed6067cbfb8c31a66795fce WatchSource:0}: Error finding container fbd5903d67fe5b47527b0e2b004c1109cc8d3f5c1ed6067cbfb8c31a66795fce: Status 404 returned error can't find the container with id fbd5903d67fe5b47527b0e2b004c1109cc8d3f5c1ed6067cbfb8c31a66795fce Jul 21 21:49:47.150741 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:47.150650 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-jhvdw" event={"ID":"80b73dfc-bb0f-4fc1-ae01-069daef10f00","Type":"ContainerStarted","Data":"fbd5903d67fe5b47527b0e2b004c1109cc8d3f5c1ed6067cbfb8c31a66795fce"} Jul 21 21:49:48.506899 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:48.506862 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret\") pod \"global-pull-secret-syncer-fb457\" (UID: \"b3de97b9-626a-4516-9749-94ff62f60273\") " pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:48.506899 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:48.506908 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs\") pod \"network-metrics-daemon-zxm6p\" (UID: \"052e4590-9613-4a30-b87c-fe56d7a11e9a\") " pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:48.507456 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:48.506943 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2qm\" (UniqueName: \"kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm\") pod \"network-check-target-zrk2g\" (UID: \"1b7dcb72-e10b-4f88-be68-79d8b3b97f87\") " pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:48.509734 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:48.509712 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 21:49:48.510587 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:48.510567 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 21:49:48.510680 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:48.510589 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 21:49:48.520254 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:48.520232 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 21:49:48.520363 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:48.520319 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/052e4590-9613-4a30-b87c-fe56d7a11e9a-metrics-certs\") pod \"network-metrics-daemon-zxm6p\" (UID: \"052e4590-9613-4a30-b87c-fe56d7a11e9a\") " pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:48.521436 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:48.521411 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3de97b9-626a-4516-9749-94ff62f60273-original-pull-secret\") pod \"global-pull-secret-syncer-fb457\" (UID: \"b3de97b9-626a-4516-9749-94ff62f60273\") " pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:48.530487 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:48.530469 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz2qm\" (UniqueName: \"kubernetes.io/projected/1b7dcb72-e10b-4f88-be68-79d8b3b97f87-kube-api-access-bz2qm\") pod \"network-check-target-zrk2g\" (UID: \"1b7dcb72-e10b-4f88-be68-79d8b3b97f87\") " pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:48.591419 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:48.591399 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fd2qt\"" Jul 21 21:49:48.598123 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:48.598106 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n9lxs\"" Jul 21 21:49:48.599164 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:48.599146 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:48.606182 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:48.606143 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxm6p" Jul 21 21:49:48.782552 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:48.782484 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fb457" Jul 21 21:49:50.703759 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:50.703680 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fb457"] Jul 21 21:49:50.707013 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:49:50.706986 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3de97b9_626a_4516_9749_94ff62f60273.slice/crio-ddb0c5262dfbd14111bc20e75df6aa99d8feedafd549c7d71209e850d3c10e23 WatchSource:0}: Error finding container ddb0c5262dfbd14111bc20e75df6aa99d8feedafd549c7d71209e850d3c10e23: Status 404 returned error can't find the container with id ddb0c5262dfbd14111bc20e75df6aa99d8feedafd549c7d71209e850d3c10e23 Jul 21 21:49:50.914154 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:50.914126 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zrk2g"] Jul 21 21:49:50.914848 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:50.914779 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zxm6p"] Jul 21 21:49:50.917584 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:49:50.917561 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052e4590_9613_4a30_b87c_fe56d7a11e9a.slice/crio-d726d46534c1e57610ab4636c1fa268787d68ad867958327722b58d47744e4cb WatchSource:0}: Error finding container d726d46534c1e57610ab4636c1fa268787d68ad867958327722b58d47744e4cb: Status 404 returned error can't find the container with id d726d46534c1e57610ab4636c1fa268787d68ad867958327722b58d47744e4cb Jul 21 21:49:50.918094 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:49:50.918063 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7dcb72_e10b_4f88_be68_79d8b3b97f87.slice/crio-b3b6ae3ea4023cec0e8107286764f4f10f54d2b2a48eb847b5116a7d623a7a0c WatchSource:0}: Error finding container b3b6ae3ea4023cec0e8107286764f4f10f54d2b2a48eb847b5116a7d623a7a0c: Status 404 returned error can't find the container with id b3b6ae3ea4023cec0e8107286764f4f10f54d2b2a48eb847b5116a7d623a7a0c Jul 21 21:49:51.164115 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.164009 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-jhvdw" event={"ID":"80b73dfc-bb0f-4fc1-ae01-069daef10f00","Type":"ContainerStarted","Data":"31a9902015dc72a28b068fb5ae1f75c75e70b8df961343c14add2c302eba513d"} Jul 21 21:49:51.164312 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.164183 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-jhvdw" Jul 21 21:49:51.165325 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.165296 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fb457" event={"ID":"b3de97b9-626a-4516-9749-94ff62f60273","Type":"ContainerStarted","Data":"ddb0c5262dfbd14111bc20e75df6aa99d8feedafd549c7d71209e850d3c10e23"} Jul 21 21:49:51.166849 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.166826 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zxm6p" event={"ID":"052e4590-9613-4a30-b87c-fe56d7a11e9a","Type":"ContainerStarted","Data":"d726d46534c1e57610ab4636c1fa268787d68ad867958327722b58d47744e4cb"} Jul 21 21:49:51.168055 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.168032 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zrk2g" event={"ID":"1b7dcb72-e10b-4f88-be68-79d8b3b97f87","Type":"ContainerStarted","Data":"b3b6ae3ea4023cec0e8107286764f4f10f54d2b2a48eb847b5116a7d623a7a0c"} Jul 21 21:49:51.169594 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.169571 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f5d79568-5vmjl" event={"ID":"6c341c9f-8559-4335-8624-82f38b0cdbeb","Type":"ContainerStarted","Data":"2a3610196e4a1e9c3a9dd8cfeb0a9ebe41e28f2e6a2cc8df5ee07470120521e7"} Jul 21 21:49:51.170069 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.170052 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-jhvdw" Jul 21 21:49:51.172517 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.172495 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pdv8n" event={"ID":"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a","Type":"ContainerStarted","Data":"7ef6fac2fae759870daac9ef97b99ceae064ac8f769d966c9ed853fdd5156531"} Jul 21 21:49:51.185219 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.184332 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-jhvdw" podStartSLOduration=2.047502478 podStartE2EDuration="6.184314855s" totalImagesPullingTime="4.136812377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:51.180906735 +0000 UTC m=+68.944141001" watchObservedRunningTime="2026-07-21 21:49:51.184314855 +0000 UTC m=+68.947549119" Jul 21 21:49:51.217343 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.217303 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54f5d79568-5vmjl" podStartSLOduration=1.5110921990000001 podStartE2EDuration="6.217289461s" totalImagesPullingTime="4.706197262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:51.216438166 +0000 UTC m=+68.979672428" watchObservedRunningTime="2026-07-21 21:49:51.217289461 +0000 UTC m=+68.980523719" Jul 21 21:49:51.359310 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.359276 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-bspgk"] Jul 21 21:49:51.373439 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.373407 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-bspgk"] Jul 21 21:49:51.373586 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.373527 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bspgk" Jul 21 21:49:51.379018 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.378982 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 21 21:49:51.379018 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.378999 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 21:49:51.379307 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.379267 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nn4rb\"" Jul 21 21:49:51.379493 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.379312 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 21:49:51.379588 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.379572 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 21:49:51.379795 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.379765 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 21 21:49:51.429084 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.428830 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/148d42ef-cb3e-4912-bea7-942efecb697a-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-bspgk\" (UID: \"148d42ef-cb3e-4912-bea7-942efecb697a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bspgk" Jul 21 21:49:51.429084 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.428890 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/148d42ef-cb3e-4912-bea7-942efecb697a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-bspgk\" (UID: \"148d42ef-cb3e-4912-bea7-942efecb697a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bspgk" Jul 21 21:49:51.429084 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.428920 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/148d42ef-cb3e-4912-bea7-942efecb697a-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-bspgk\" (UID: \"148d42ef-cb3e-4912-bea7-942efecb697a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bspgk" Jul 21 21:49:51.429084 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.428950 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m86lg\" (UniqueName: \"kubernetes.io/projected/148d42ef-cb3e-4912-bea7-942efecb697a-kube-api-access-m86lg\") pod \"prometheus-operator-5945b5cfd-bspgk\" (UID: \"148d42ef-cb3e-4912-bea7-942efecb697a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bspgk" Jul 21 21:49:51.530937 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.529275 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/148d42ef-cb3e-4912-bea7-942efecb697a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-bspgk\" (UID: \"148d42ef-cb3e-4912-bea7-942efecb697a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bspgk" Jul 21 21:49:51.530937 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.529319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/148d42ef-cb3e-4912-bea7-942efecb697a-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-bspgk\" (UID: \"148d42ef-cb3e-4912-bea7-942efecb697a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bspgk" Jul 21 21:49:51.530937 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.529349 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m86lg\" (UniqueName: \"kubernetes.io/projected/148d42ef-cb3e-4912-bea7-942efecb697a-kube-api-access-m86lg\") pod \"prometheus-operator-5945b5cfd-bspgk\" (UID: \"148d42ef-cb3e-4912-bea7-942efecb697a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bspgk" Jul 21 21:49:51.530937 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.529437 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/148d42ef-cb3e-4912-bea7-942efecb697a-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-bspgk\" (UID: \"148d42ef-cb3e-4912-bea7-942efecb697a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bspgk" Jul 21 21:49:51.530937 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.530623 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/148d42ef-cb3e-4912-bea7-942efecb697a-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-bspgk\" (UID: \"148d42ef-cb3e-4912-bea7-942efecb697a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bspgk" Jul 21 21:49:51.535275 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.535216 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/148d42ef-cb3e-4912-bea7-942efecb697a-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-bspgk\" (UID: \"148d42ef-cb3e-4912-bea7-942efecb697a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bspgk" Jul 21 21:49:51.536496 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.536476 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/148d42ef-cb3e-4912-bea7-942efecb697a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-bspgk\" (UID: \"148d42ef-cb3e-4912-bea7-942efecb697a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bspgk" Jul 21 21:49:51.546940 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.546898 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m86lg\" (UniqueName: \"kubernetes.io/projected/148d42ef-cb3e-4912-bea7-942efecb697a-kube-api-access-m86lg\") pod \"prometheus-operator-5945b5cfd-bspgk\" (UID: \"148d42ef-cb3e-4912-bea7-942efecb697a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bspgk" Jul 21 21:49:51.688579 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.688491 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bspgk" Jul 21 21:49:51.873928 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:51.873848 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-bspgk"] Jul 21 21:49:52.319890 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:49:52.319848 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148d42ef_cb3e_4912_bea7_942efecb697a.slice/crio-40d7e1266501c791cf76c8dfa62b27a6ed1a5e26d7f259a001a1358a38be3b72 WatchSource:0}: Error finding container 40d7e1266501c791cf76c8dfa62b27a6ed1a5e26d7f259a001a1358a38be3b72: Status 404 returned error can't find the container with id 40d7e1266501c791cf76c8dfa62b27a6ed1a5e26d7f259a001a1358a38be3b72 Jul 21 21:49:53.187941 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:53.187899 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zxm6p" event={"ID":"052e4590-9613-4a30-b87c-fe56d7a11e9a","Type":"ContainerStarted","Data":"fd66da3a65fe62ceeec8a2b6e87a370db0d271941a13e506a8517ce33dedaaa9"} Jul 21 21:49:53.188749 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:53.188580 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zxm6p" event={"ID":"052e4590-9613-4a30-b87c-fe56d7a11e9a","Type":"ContainerStarted","Data":"b82d0b15a1067018404d8f448e54e00731025db616c1ab1aed71e4605bd4a9bb"} Jul 21 21:49:53.190840 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:53.190789 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bspgk" event={"ID":"148d42ef-cb3e-4912-bea7-942efecb697a","Type":"ContainerStarted","Data":"40d7e1266501c791cf76c8dfa62b27a6ed1a5e26d7f259a001a1358a38be3b72"} Jul 21 21:49:53.206461 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:53.205831 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zxm6p" podStartSLOduration=68.439350088 podStartE2EDuration="1m10.20581501s" totalImagesPullingTime="1.766464922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:53.204224732 +0000 UTC m=+70.967458995" watchObservedRunningTime="2026-07-21 21:49:53.20581501 +0000 UTC m=+70.969049274" Jul 21 21:49:55.649454 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:55.649416 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:55.649991 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:55.649486 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:55.655802 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:55.655778 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:56.153329 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:56.153297 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5l4xt" Jul 21 21:49:56.207308 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:56.207279 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:49:58.212040 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:58.211957 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zrk2g" event={"ID":"1b7dcb72-e10b-4f88-be68-79d8b3b97f87","Type":"ContainerStarted","Data":"d5b3fd222167056091d41fa2334bf4df123be59f7497d41942d8470021962025"} Jul 21 21:49:58.212494 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:58.212075 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:49:58.214132 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:58.214103 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pdv8n" event={"ID":"6eb0fb30-a50c-4913-b8e0-907e9fc18c7a","Type":"ContainerStarted","Data":"d0a4a7af1c1db80b71a0ac7d63e15c2d7a5f023eeb6afd05bff08343a41146a6"} Jul 21 21:49:58.215552 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:58.215522 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fb457" event={"ID":"b3de97b9-626a-4516-9749-94ff62f60273","Type":"ContainerStarted","Data":"df83d5ebd625a0f11d1d52c5b28f17b12de66b6495c04ec914796c0eab968572"} Jul 21 21:49:58.217170 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:58.217148 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bspgk" event={"ID":"148d42ef-cb3e-4912-bea7-942efecb697a","Type":"ContainerStarted","Data":"5ced921e067bd0470708df82738beafc7ba595558e644e2c284f45b77ef84586"} Jul 21 21:49:58.217270 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:58.217183 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bspgk" event={"ID":"148d42ef-cb3e-4912-bea7-942efecb697a","Type":"ContainerStarted","Data":"9b4b9186bbd1d8d3e55543ebaf7019dfb83854ba58b38519cf8f439abbcd31be"} Jul 21 21:49:58.234967 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:58.234915 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zrk2g" podStartSLOduration=68.316380106 podStartE2EDuration="1m15.234902241s" totalImagesPullingTime="6.918522135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:58.234439606 +0000 UTC m=+75.997673861" watchObservedRunningTime="2026-07-21 21:49:58.234902241 +0000 UTC m=+75.998136502" Jul 21 21:49:58.257210 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:58.257150 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fb457" podStartSLOduration=68.118656881 podStartE2EDuration="1m15.257133495s" totalImagesPullingTime="7.138476614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:58.256568249 +0000 UTC m=+76.019802514" watchObservedRunningTime="2026-07-21 21:49:58.257133495 +0000 UTC m=+76.020367747" Jul 21 21:49:58.279054 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:58.279002 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pdv8n" podStartSLOduration=1.17741624 podStartE2EDuration="19.278986405s" totalImagesPullingTime="18.101570165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:49:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:58.278056151 +0000 UTC m=+76.041290418" watchObservedRunningTime="2026-07-21 21:49:58.278986405 +0000 UTC m=+76.042220665" Jul 21 21:49:58.296552 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:58.296507 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bspgk" podStartSLOduration=1.788027451 podStartE2EDuration="7.296492294s" totalImagesPullingTime="5.508464843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:58.295115391 +0000 UTC m=+76.058349653" watchObservedRunningTime="2026-07-21 21:49:58.296492294 +0000 UTC m=+76.059726555" Jul 21 21:49:59.740613 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.740580 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t"] Jul 21 21:49:59.743716 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.743698 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" Jul 21 21:49:59.746332 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.746307 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 21 21:49:59.746532 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.746407 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vqs4d\"" Jul 21 21:49:59.746613 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.746581 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 21 21:49:59.755502 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.755480 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t"] Jul 21 21:49:59.764427 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.764402 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ct72r"] Jul 21 21:49:59.767733 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.767715 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-jtbpf"] Jul 21 21:49:59.767853 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.767839 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:49:59.770570 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.770549 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 21:49:59.770570 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.770564 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7blhv\"" Jul 21 21:49:59.770740 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.770716 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:49:59.770832 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.770816 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 21:49:59.770906 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.770882 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 21:49:59.772879 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.772865 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qrzrq\"" Jul 21 21:49:59.773201 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.773185 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 21 21:49:59.773847 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.773822 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 21 21:49:59.773934 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.773917 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 21 21:49:59.785904 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.785857 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-jtbpf"] Jul 21 21:49:59.899701 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.899663 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kwd7\" (UniqueName: \"kubernetes.io/projected/3a164408-f1fb-46ab-9ae2-54216f57825e-kube-api-access-5kwd7\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:49:59.899701 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.899704 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3a164408-f1fb-46ab-9ae2-54216f57825e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:49:59.899915 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.899723 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/908476c2-8321-46b7-b5b2-bc0d833e232c-node-exporter-accelerators-collector-config\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:49:59.899915 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.899746 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7b0ae59-42f4-4d6f-a389-a23f01a1e718-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-fz57t\" (UID: \"b7b0ae59-42f4-4d6f-a389-a23f01a1e718\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" Jul 21 21:49:59.899915 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.899825 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/908476c2-8321-46b7-b5b2-bc0d833e232c-node-exporter-wtmp\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:49:59.899915 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.899862 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/908476c2-8321-46b7-b5b2-bc0d833e232c-root\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:49:59.899915 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.899880 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/908476c2-8321-46b7-b5b2-bc0d833e232c-node-exporter-tls\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:49:59.900074 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.899917 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7b0ae59-42f4-4d6f-a389-a23f01a1e718-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-fz57t\" (UID: \"b7b0ae59-42f4-4d6f-a389-a23f01a1e718\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" Jul 21 21:49:59.900074 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.899936 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/908476c2-8321-46b7-b5b2-bc0d833e232c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:49:59.900074 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.899952 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mzsc\" (UniqueName: \"kubernetes.io/projected/908476c2-8321-46b7-b5b2-bc0d833e232c-kube-api-access-4mzsc\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:49:59.900074 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.899970 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a164408-f1fb-46ab-9ae2-54216f57825e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:49:59.900074 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.899994 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3a164408-f1fb-46ab-9ae2-54216f57825e-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:49:59.900074 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.900012 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qfrc\" (UniqueName: \"kubernetes.io/projected/b7b0ae59-42f4-4d6f-a389-a23f01a1e718-kube-api-access-9qfrc\") pod \"openshift-state-metrics-79f6958cdf-fz57t\" (UID: \"b7b0ae59-42f4-4d6f-a389-a23f01a1e718\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" Jul 21 21:49:59.900074 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.900029 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7b0ae59-42f4-4d6f-a389-a23f01a1e718-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-fz57t\" (UID: \"b7b0ae59-42f4-4d6f-a389-a23f01a1e718\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" Jul 21 21:49:59.900074 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.900054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a164408-f1fb-46ab-9ae2-54216f57825e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:49:59.900310 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.900089 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/908476c2-8321-46b7-b5b2-bc0d833e232c-metrics-client-ca\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:49:59.900310 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.900123 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/908476c2-8321-46b7-b5b2-bc0d833e232c-sys\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:49:59.900310 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.900143 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/908476c2-8321-46b7-b5b2-bc0d833e232c-node-exporter-textfile\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:49:59.900310 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:49:59.900187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a164408-f1fb-46ab-9ae2-54216f57825e-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:50:00.001112 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001009 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/908476c2-8321-46b7-b5b2-bc0d833e232c-node-exporter-accelerators-collector-config\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.001112 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001068 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7b0ae59-42f4-4d6f-a389-a23f01a1e718-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-fz57t\" (UID: \"b7b0ae59-42f4-4d6f-a389-a23f01a1e718\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" Jul 21 21:50:00.001112 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/908476c2-8321-46b7-b5b2-bc0d833e232c-node-exporter-wtmp\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.001112 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001117 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/908476c2-8321-46b7-b5b2-bc0d833e232c-root\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.001486 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001135 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/908476c2-8321-46b7-b5b2-bc0d833e232c-node-exporter-tls\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.001486 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001160 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7b0ae59-42f4-4d6f-a389-a23f01a1e718-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-fz57t\" (UID: \"b7b0ae59-42f4-4d6f-a389-a23f01a1e718\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" Jul 21 21:50:00.001486 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001185 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/908476c2-8321-46b7-b5b2-bc0d833e232c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.001486 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001212 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mzsc\" (UniqueName: \"kubernetes.io/projected/908476c2-8321-46b7-b5b2-bc0d833e232c-kube-api-access-4mzsc\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.001486 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001217 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/908476c2-8321-46b7-b5b2-bc0d833e232c-root\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.001486 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a164408-f1fb-46ab-9ae2-54216f57825e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:50:00.001486 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001279 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/908476c2-8321-46b7-b5b2-bc0d833e232c-node-exporter-wtmp\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.001486 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:50:00.001286 2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 21 21:50:00.001486 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:50:00.001294 2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 21 21:50:00.001486 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:50:00.001401 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/908476c2-8321-46b7-b5b2-bc0d833e232c-node-exporter-tls podName:908476c2-8321-46b7-b5b2-bc0d833e232c nodeName:}" failed. No retries permitted until 2026-07-21 21:50:00.501350333 +0000 UTC m=+78.264584572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/908476c2-8321-46b7-b5b2-bc0d833e232c-node-exporter-tls") pod "node-exporter-ct72r" (UID: "908476c2-8321-46b7-b5b2-bc0d833e232c") : secret "node-exporter-tls" not found Jul 21 21:50:00.001486 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:50:00.001449 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b0ae59-42f4-4d6f-a389-a23f01a1e718-openshift-state-metrics-tls podName:b7b0ae59-42f4-4d6f-a389-a23f01a1e718 nodeName:}" failed. No retries permitted until 2026-07-21 21:50:00.501431082 +0000 UTC m=+78.264665322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b7b0ae59-42f4-4d6f-a389-a23f01a1e718-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-fz57t" (UID: "b7b0ae59-42f4-4d6f-a389-a23f01a1e718") : secret "openshift-state-metrics-tls" not found Jul 21 21:50:00.002070 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3a164408-f1fb-46ab-9ae2-54216f57825e-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:50:00.002070 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qfrc\" (UniqueName: \"kubernetes.io/projected/b7b0ae59-42f4-4d6f-a389-a23f01a1e718-kube-api-access-9qfrc\") pod \"openshift-state-metrics-79f6958cdf-fz57t\" (UID: \"b7b0ae59-42f4-4d6f-a389-a23f01a1e718\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" Jul 21 21:50:00.002070 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001561 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7b0ae59-42f4-4d6f-a389-a23f01a1e718-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-fz57t\" (UID: \"b7b0ae59-42f4-4d6f-a389-a23f01a1e718\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" Jul 21 21:50:00.002070 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001592 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a164408-f1fb-46ab-9ae2-54216f57825e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:50:00.002070 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001619 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/908476c2-8321-46b7-b5b2-bc0d833e232c-metrics-client-ca\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.002070 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/908476c2-8321-46b7-b5b2-bc0d833e232c-sys\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.002070 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/908476c2-8321-46b7-b5b2-bc0d833e232c-node-exporter-textfile\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.002070 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001704 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a164408-f1fb-46ab-9ae2-54216f57825e-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:50:00.002070 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001751 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kwd7\" (UniqueName: \"kubernetes.io/projected/3a164408-f1fb-46ab-9ae2-54216f57825e-kube-api-access-5kwd7\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:50:00.002070 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3a164408-f1fb-46ab-9ae2-54216f57825e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:50:00.002070 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001851 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/908476c2-8321-46b7-b5b2-bc0d833e232c-node-exporter-accelerators-collector-config\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.002070 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001865 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3a164408-f1fb-46ab-9ae2-54216f57825e-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:50:00.002070 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.001913 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/908476c2-8321-46b7-b5b2-bc0d833e232c-sys\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.002070 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:50:00.001984 2580 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 21 21:50:00.002070 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:50:00.002026 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a164408-f1fb-46ab-9ae2-54216f57825e-kube-state-metrics-tls podName:3a164408-f1fb-46ab-9ae2-54216f57825e nodeName:}" failed. No retries permitted until 2026-07-21 21:50:00.502010964 +0000 UTC m=+78.265245208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3a164408-f1fb-46ab-9ae2-54216f57825e-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-jtbpf" (UID: "3a164408-f1fb-46ab-9ae2-54216f57825e") : secret "kube-state-metrics-tls" not found Jul 21 21:50:00.002070 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.002051 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7b0ae59-42f4-4d6f-a389-a23f01a1e718-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-fz57t\" (UID: \"b7b0ae59-42f4-4d6f-a389-a23f01a1e718\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" Jul 21 21:50:00.002988 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.002451 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/908476c2-8321-46b7-b5b2-bc0d833e232c-node-exporter-textfile\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.002988 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.002478 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/908476c2-8321-46b7-b5b2-bc0d833e232c-metrics-client-ca\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.002988 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.002487 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3a164408-f1fb-46ab-9ae2-54216f57825e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:50:00.002988 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.002745 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a164408-f1fb-46ab-9ae2-54216f57825e-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:50:00.004366 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.004342 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/908476c2-8321-46b7-b5b2-bc0d833e232c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.004496 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.004456 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7b0ae59-42f4-4d6f-a389-a23f01a1e718-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-fz57t\" (UID: \"b7b0ae59-42f4-4d6f-a389-a23f01a1e718\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" Jul 21 21:50:00.004496 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.004459 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a164408-f1fb-46ab-9ae2-54216f57825e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:50:00.027207 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.027177 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kwd7\" (UniqueName: \"kubernetes.io/projected/3a164408-f1fb-46ab-9ae2-54216f57825e-kube-api-access-5kwd7\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:50:00.029590 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.029563 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mzsc\" (UniqueName: \"kubernetes.io/projected/908476c2-8321-46b7-b5b2-bc0d833e232c-kube-api-access-4mzsc\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.034503 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.034478 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qfrc\" (UniqueName: \"kubernetes.io/projected/b7b0ae59-42f4-4d6f-a389-a23f01a1e718-kube-api-access-9qfrc\") pod \"openshift-state-metrics-79f6958cdf-fz57t\" (UID: \"b7b0ae59-42f4-4d6f-a389-a23f01a1e718\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" Jul 21 21:50:00.507112 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.507076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/908476c2-8321-46b7-b5b2-bc0d833e232c-node-exporter-tls\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.507112 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.507116 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7b0ae59-42f4-4d6f-a389-a23f01a1e718-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-fz57t\" (UID: \"b7b0ae59-42f4-4d6f-a389-a23f01a1e718\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" Jul 21 21:50:00.507423 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.507144 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a164408-f1fb-46ab-9ae2-54216f57825e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:50:00.509618 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.509593 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/908476c2-8321-46b7-b5b2-bc0d833e232c-node-exporter-tls\") pod \"node-exporter-ct72r\" (UID: \"908476c2-8321-46b7-b5b2-bc0d833e232c\") " pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.509741 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.509724 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a164408-f1fb-46ab-9ae2-54216f57825e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jtbpf\" (UID: \"3a164408-f1fb-46ab-9ae2-54216f57825e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:50:00.509780 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.509731 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7b0ae59-42f4-4d6f-a389-a23f01a1e718-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-fz57t\" (UID: \"b7b0ae59-42f4-4d6f-a389-a23f01a1e718\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" Jul 21 21:50:00.652211 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.652176 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" Jul 21 21:50:00.678087 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.678060 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ct72r" Jul 21 21:50:00.687988 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.687967 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" Jul 21 21:50:00.691785 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:50:00.691517 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908476c2_8321_46b7_b5b2_bc0d833e232c.slice/crio-ac0bfc62af2ded33e8faab9ff70634efd7d4a96de934464ff3dfc70891fd0eea WatchSource:0}: Error finding container ac0bfc62af2ded33e8faab9ff70634efd7d4a96de934464ff3dfc70891fd0eea: Status 404 returned error can't find the container with id ac0bfc62af2ded33e8faab9ff70634efd7d4a96de934464ff3dfc70891fd0eea Jul 21 21:50:00.744577 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.744363 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 21:50:00.756081 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.756046 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:00.759149 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.758858 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 21:50:00.760819 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.760573 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 21 21:50:00.760930 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.760828 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 21 21:50:00.762939 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.761028 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 21 21:50:00.762939 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.761296 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 21 21:50:00.762939 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.761529 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 21 21:50:00.762939 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.761764 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-f72k6\"" Jul 21 21:50:00.762939 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.761979 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 21 21:50:00.762939 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.762172 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 21 21:50:00.762939 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.762352 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 21 21:50:00.763338 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.763316 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 21 21:50:00.807923 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.807843 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t"] Jul 21 21:50:00.849802 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.849661 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-jtbpf"] Jul 21 21:50:00.851740 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:50:00.851716 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a164408_f1fb_46ab_9ae2_54216f57825e.slice/crio-fa3369cf4dd74522982bf748e678a03d63163f22781fdbd46df2c75b0d8a0500 WatchSource:0}: Error finding container fa3369cf4dd74522982bf748e678a03d63163f22781fdbd46df2c75b0d8a0500: Status 404 returned error can't find the container with id fa3369cf4dd74522982bf748e678a03d63163f22781fdbd46df2c75b0d8a0500 Jul 21 21:50:00.911098 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.911074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:00.911230 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.911204 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-config-volume\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:00.911279 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.911236 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-config-out\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:00.911279 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.911254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-web-config\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:00.911279 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.911275 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:00.911410 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.911320 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:00.911410 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.911347 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:00.911410 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.911364 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp7r7\" (UniqueName: \"kubernetes.io/projected/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-kube-api-access-rp7r7\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:00.911410 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.911401 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:00.911559 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.911423 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:00.911559 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.911441 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:00.911559 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.911458 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:00.911559 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:00.911524 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.012178 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.012090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.012178 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.012130 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-config-volume\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.012178 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.012156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-config-out\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.012178 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.012175 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-web-config\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.012513 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.012289 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.012513 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.012338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.012513 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.012401 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.012842 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.012806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp7r7\" (UniqueName: \"kubernetes.io/projected/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-kube-api-access-rp7r7\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.012928 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.012854 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.012928 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.012896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.013032 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.012930 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.013032 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.012963 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.013032 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.012992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.013352 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.013175 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.013492 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.013471 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.014367 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.014345 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.015343 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.015296 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.015754 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.015628 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-config-volume\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.015941 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.015899 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.016060 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.015947 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-web-config\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.016627 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.016600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.017033 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.017018 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-config-out\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.017562 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.017542 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.017637 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.017541 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.018472 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.018450 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.021920 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.021900 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp7r7\" (UniqueName: \"kubernetes.io/projected/7f8ecf16-5c8e-48e1-bebc-cc747db6dd91-kube-api-access-rp7r7\") pod \"alertmanager-main-0\" (UID: \"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.074945 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.074916 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 21:50:01.215041 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.215012 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 21:50:01.217749 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:50:01.217725 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8ecf16_5c8e_48e1_bebc_cc747db6dd91.slice/crio-d44a0edaa6873672947b14a25750737ec38db56739d157672481e94565c93f73 WatchSource:0}: Error finding container d44a0edaa6873672947b14a25750737ec38db56739d157672481e94565c93f73: Status 404 returned error can't find the container with id d44a0edaa6873672947b14a25750737ec38db56739d157672481e94565c93f73 Jul 21 21:50:01.233238 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.233204 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" event={"ID":"b7b0ae59-42f4-4d6f-a389-a23f01a1e718","Type":"ContainerStarted","Data":"363bf9a3912eff290c2e95daa8103d9ba3d8f55b5237994d540d94ba48c437a0"} Jul 21 21:50:01.233360 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.233277 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" event={"ID":"b7b0ae59-42f4-4d6f-a389-a23f01a1e718","Type":"ContainerStarted","Data":"8e5fc4824f562261ee0dc7b01c14b895031b4c3b0c3a38350f7672084cda6473"} Jul 21 21:50:01.233360 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.233313 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" event={"ID":"b7b0ae59-42f4-4d6f-a389-a23f01a1e718","Type":"ContainerStarted","Data":"7d098083a6676574b734c65862b7ed0618bafa97df785520f7db73dd9dd1fd6e"} Jul 21 21:50:01.234448 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.234361 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91","Type":"ContainerStarted","Data":"d44a0edaa6873672947b14a25750737ec38db56739d157672481e94565c93f73"} Jul 21 21:50:01.235450 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.235431 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" event={"ID":"3a164408-f1fb-46ab-9ae2-54216f57825e","Type":"ContainerStarted","Data":"fa3369cf4dd74522982bf748e678a03d63163f22781fdbd46df2c75b0d8a0500"} Jul 21 21:50:01.236429 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.236396 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ct72r" event={"ID":"908476c2-8321-46b7-b5b2-bc0d833e232c","Type":"ContainerStarted","Data":"ac0bfc62af2ded33e8faab9ff70634efd7d4a96de934464ff3dfc70891fd0eea"} Jul 21 21:50:01.773169 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.773131 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-644856656-r8znz"] Jul 21 21:50:01.778218 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.778186 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:01.781295 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.781267 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 21 21:50:01.781460 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.781420 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 21 21:50:01.781460 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.781420 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9nbc7\"" Jul 21 21:50:01.781584 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.781474 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-baboqicc1lpht\"" Jul 21 21:50:01.781584 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.781254 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 21 21:50:01.781678 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.781614 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 21 21:50:01.781840 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.781819 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 21 21:50:01.790090 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.789994 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-644856656-r8znz"] Jul 21 21:50:01.921234 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.921198 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/860ea5df-15b1-473d-a82b-fb8b89136cee-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:01.921414 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.921240 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/860ea5df-15b1-473d-a82b-fb8b89136cee-secret-thanos-querier-tls\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:01.921414 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.921276 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/860ea5df-15b1-473d-a82b-fb8b89136cee-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:01.921414 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.921366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/860ea5df-15b1-473d-a82b-fb8b89136cee-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:01.921576 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.921483 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dddx\" (UniqueName: \"kubernetes.io/projected/860ea5df-15b1-473d-a82b-fb8b89136cee-kube-api-access-8dddx\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:01.921576 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.921529 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/860ea5df-15b1-473d-a82b-fb8b89136cee-metrics-client-ca\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:01.921576 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.921558 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/860ea5df-15b1-473d-a82b-fb8b89136cee-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:01.921709 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:01.921617 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/860ea5df-15b1-473d-a82b-fb8b89136cee-secret-grpc-tls\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:02.023062 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:02.023025 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/860ea5df-15b1-473d-a82b-fb8b89136cee-metrics-client-ca\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:02.023280 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:02.023070 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/860ea5df-15b1-473d-a82b-fb8b89136cee-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:02.023280 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:02.023108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/860ea5df-15b1-473d-a82b-fb8b89136cee-secret-grpc-tls\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:02.023280 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:02.023155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/860ea5df-15b1-473d-a82b-fb8b89136cee-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:02.023280 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:02.023182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/860ea5df-15b1-473d-a82b-fb8b89136cee-secret-thanos-querier-tls\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:02.023280 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:02.023210 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/860ea5df-15b1-473d-a82b-fb8b89136cee-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:02.023280 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:02.023240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/860ea5df-15b1-473d-a82b-fb8b89136cee-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:02.023597 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:02.023296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dddx\" (UniqueName: \"kubernetes.io/projected/860ea5df-15b1-473d-a82b-fb8b89136cee-kube-api-access-8dddx\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:02.023852 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:02.023826 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/860ea5df-15b1-473d-a82b-fb8b89136cee-metrics-client-ca\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:02.026687 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:02.026660 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/860ea5df-15b1-473d-a82b-fb8b89136cee-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:02.026801 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:02.026660 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/860ea5df-15b1-473d-a82b-fb8b89136cee-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:02.027093 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:02.027070 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/860ea5df-15b1-473d-a82b-fb8b89136cee-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:02.027225 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:02.027200 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/860ea5df-15b1-473d-a82b-fb8b89136cee-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:02.027274 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:02.027234 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/860ea5df-15b1-473d-a82b-fb8b89136cee-secret-thanos-querier-tls\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:02.027333 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:02.027316 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/860ea5df-15b1-473d-a82b-fb8b89136cee-secret-grpc-tls\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:02.035145 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:02.035121 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dddx\" (UniqueName: \"kubernetes.io/projected/860ea5df-15b1-473d-a82b-fb8b89136cee-kube-api-access-8dddx\") pod \"thanos-querier-644856656-r8znz\" (UID: \"860ea5df-15b1-473d-a82b-fb8b89136cee\") " pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:02.091025 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:02.090985 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:02.704065 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:02.704035 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-644856656-r8znz"] Jul 21 21:50:02.842774 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:50:02.842700 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860ea5df_15b1_473d_a82b_fb8b89136cee.slice/crio-c3cff894213e67b4dd90e5446e0f20d532bb9d83dbac77a01b670a9aa9cdca0d WatchSource:0}: Error finding container c3cff894213e67b4dd90e5446e0f20d532bb9d83dbac77a01b670a9aa9cdca0d: Status 404 returned error can't find the container with id c3cff894213e67b4dd90e5446e0f20d532bb9d83dbac77a01b670a9aa9cdca0d Jul 21 21:50:03.249111 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:03.248977 2580 generic.go:356] "Generic (PLEG): container finished" podID="7f8ecf16-5c8e-48e1-bebc-cc747db6dd91" containerID="c71b84a3cf4edbde23b30de5ee19e5b730c70dad1e60aa9f88ec5bb26c96a99d" exitCode=0 Jul 21 21:50:03.249111 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:03.249067 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91","Type":"ContainerDied","Data":"c71b84a3cf4edbde23b30de5ee19e5b730c70dad1e60aa9f88ec5bb26c96a99d"} Jul 21 21:50:03.251386 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:03.251349 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" event={"ID":"3a164408-f1fb-46ab-9ae2-54216f57825e","Type":"ContainerStarted","Data":"196f70000985cd3a41a0fea7ebaefc85b139821823af4ebf9a32ce2c80294a37"} Jul 21 21:50:03.251483 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:03.251419 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" event={"ID":"3a164408-f1fb-46ab-9ae2-54216f57825e","Type":"ContainerStarted","Data":"ba92802a0121c60d40bac4b33823c4df137302fe391be5b62f97ba22a0a27018"} Jul 21 21:50:03.251483 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:03.251452 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" event={"ID":"3a164408-f1fb-46ab-9ae2-54216f57825e","Type":"ContainerStarted","Data":"6c4f5e18cbbb60e5f3ea239579c1b67cf9f6b4015b9a6d80950bb57c0e165fef"} Jul 21 21:50:03.253230 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:03.253210 2580 generic.go:356] "Generic (PLEG): container finished" podID="908476c2-8321-46b7-b5b2-bc0d833e232c" containerID="2ed35448423612d28d4cb07a3e64e21044b6cfe17ec8fbdb1dcd12cf2692d9ae" exitCode=0 Jul 21 21:50:03.253317 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:03.253294 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ct72r" event={"ID":"908476c2-8321-46b7-b5b2-bc0d833e232c","Type":"ContainerDied","Data":"2ed35448423612d28d4cb07a3e64e21044b6cfe17ec8fbdb1dcd12cf2692d9ae"} Jul 21 21:50:03.256044 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:03.255651 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" event={"ID":"b7b0ae59-42f4-4d6f-a389-a23f01a1e718","Type":"ContainerStarted","Data":"b10b6bf0e15878fe22d7490bb262b1d4a91c613c9e42d022a0a9c02d2df1b39f"} Jul 21 21:50:03.257185 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:03.257162 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-644856656-r8znz" event={"ID":"860ea5df-15b1-473d-a82b-fb8b89136cee","Type":"ContainerStarted","Data":"c3cff894213e67b4dd90e5446e0f20d532bb9d83dbac77a01b670a9aa9cdca0d"} Jul 21 21:50:03.301402 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:03.301331 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-fz57t" podStartSLOduration=2.360468263 podStartE2EDuration="4.301316034s" totalImagesPullingTime="1.940847771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:50:03.299823329 +0000 UTC m=+81.063057610" watchObservedRunningTime="2026-07-21 21:50:03.301316034 +0000 UTC m=+81.064550295" Jul 21 21:50:03.340363 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:03.340315 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-jtbpf" podStartSLOduration=2.331036452 podStartE2EDuration="4.340298648s" totalImagesPullingTime="2.009262196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:50:03.338949638 +0000 UTC m=+81.102183918" watchObservedRunningTime="2026-07-21 21:50:03.340298648 +0000 UTC m=+81.103532910" Jul 21 21:50:04.190062 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.189546 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-8645679d69-f7ndk"] Jul 21 21:50:04.193257 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.193230 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.196079 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.196057 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 21 21:50:04.196567 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.196549 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 21 21:50:04.197408 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.197181 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 21 21:50:04.197499 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.197419 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 21 21:50:04.197657 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.197642 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dp66w\"" Jul 21 21:50:04.197980 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.197827 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4uf6fnhp202se\"" Jul 21 21:50:04.209322 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.209298 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8645679d69-f7ndk"] Jul 21 21:50:04.267108 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.267073 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ct72r" event={"ID":"908476c2-8321-46b7-b5b2-bc0d833e232c","Type":"ContainerStarted","Data":"18c5fc2f32a22c552808349782405b45a8c0ce8f6ccb96a25bd8415404599e09"} Jul 21 21:50:04.267286 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.267138 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ct72r" event={"ID":"908476c2-8321-46b7-b5b2-bc0d833e232c","Type":"ContainerStarted","Data":"fa30b2736af71b9db6382fb7fe9433912f6c99d0575997c33f1b52e011911c80"} Jul 21 21:50:04.288309 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.288246 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ct72r" podStartSLOduration=3.070360362 podStartE2EDuration="5.288227284s" totalImagesPullingTime="2.124998591s" totalInitContainerRuntime="92.868331ms" isStatefulPod=true podCreationTimestamp="2026-07-21 21:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:50:04.287177869 +0000 UTC m=+82.050412130" watchObservedRunningTime="2026-07-21 21:50:04.288227284 +0000 UTC m=+82.051461582" Jul 21 21:50:04.343349 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.343311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/19851a0f-4eac-44f8-b73c-dfbc747236cf-secret-metrics-server-client-certs\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.343531 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.343407 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwb2t\" (UniqueName: \"kubernetes.io/projected/19851a0f-4eac-44f8-b73c-dfbc747236cf-kube-api-access-lwb2t\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.343612 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.343554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/19851a0f-4eac-44f8-b73c-dfbc747236cf-metrics-server-audit-profiles\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.343690 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.343669 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/19851a0f-4eac-44f8-b73c-dfbc747236cf-client-ca-bundle\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.343751 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.343717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/19851a0f-4eac-44f8-b73c-dfbc747236cf-secret-metrics-server-tls\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.343804 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.343753 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19851a0f-4eac-44f8-b73c-dfbc747236cf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.344024 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.344003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/19851a0f-4eac-44f8-b73c-dfbc747236cf-audit-log\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.444740 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.444639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/19851a0f-4eac-44f8-b73c-dfbc747236cf-secret-metrics-server-client-certs\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.444740 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.444708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwb2t\" (UniqueName: \"kubernetes.io/projected/19851a0f-4eac-44f8-b73c-dfbc747236cf-kube-api-access-lwb2t\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.444966 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.444746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/19851a0f-4eac-44f8-b73c-dfbc747236cf-metrics-server-audit-profiles\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.444966 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.444776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/19851a0f-4eac-44f8-b73c-dfbc747236cf-client-ca-bundle\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.444966 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.444809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/19851a0f-4eac-44f8-b73c-dfbc747236cf-secret-metrics-server-tls\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.444966 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.444842 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19851a0f-4eac-44f8-b73c-dfbc747236cf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.444966 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.444872 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/19851a0f-4eac-44f8-b73c-dfbc747236cf-audit-log\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.445817 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.445780 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19851a0f-4eac-44f8-b73c-dfbc747236cf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.445944 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.445913 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/19851a0f-4eac-44f8-b73c-dfbc747236cf-metrics-server-audit-profiles\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.446325 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.446304 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/19851a0f-4eac-44f8-b73c-dfbc747236cf-audit-log\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.447759 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.447735 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/19851a0f-4eac-44f8-b73c-dfbc747236cf-client-ca-bundle\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.447972 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.447953 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/19851a0f-4eac-44f8-b73c-dfbc747236cf-secret-metrics-server-client-certs\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.448585 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.448566 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/19851a0f-4eac-44f8-b73c-dfbc747236cf-secret-metrics-server-tls\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.453172 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.453152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwb2t\" (UniqueName: \"kubernetes.io/projected/19851a0f-4eac-44f8-b73c-dfbc747236cf-kube-api-access-lwb2t\") pod \"metrics-server-8645679d69-f7ndk\" (UID: \"19851a0f-4eac-44f8-b73c-dfbc747236cf\") " pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.508397 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.508343 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:04.524969 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.524938 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tgl5d"] Jul 21 21:50:04.528238 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.528212 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tgl5d" Jul 21 21:50:04.531831 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.531804 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-b996n\"" Jul 21 21:50:04.531971 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.531950 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 21 21:50:04.538431 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.538406 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tgl5d"] Jul 21 21:50:04.646680 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.646640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cf08e0a6-5fee-4612-9805-3c91164df632-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-tgl5d\" (UID: \"cf08e0a6-5fee-4612-9805-3c91164df632\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tgl5d" Jul 21 21:50:04.646874 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.646792 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-456cn\" (UniqueName: \"kubernetes.io/projected/cf08e0a6-5fee-4612-9805-3c91164df632-kube-api-access-456cn\") pod \"monitoring-plugin-7b7b7bc8bf-tgl5d\" (UID: \"cf08e0a6-5fee-4612-9805-3c91164df632\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tgl5d" Jul 21 21:50:04.748304 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.748271 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-456cn\" (UniqueName: \"kubernetes.io/projected/cf08e0a6-5fee-4612-9805-3c91164df632-kube-api-access-456cn\") pod \"monitoring-plugin-7b7b7bc8bf-tgl5d\" (UID: \"cf08e0a6-5fee-4612-9805-3c91164df632\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tgl5d" Jul 21 21:50:04.748512 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.748315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cf08e0a6-5fee-4612-9805-3c91164df632-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-tgl5d\" (UID: \"cf08e0a6-5fee-4612-9805-3c91164df632\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tgl5d" Jul 21 21:50:04.748592 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:50:04.748514 2580 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 21 21:50:04.748644 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:50:04.748595 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf08e0a6-5fee-4612-9805-3c91164df632-monitoring-plugin-cert podName:cf08e0a6-5fee-4612-9805-3c91164df632 nodeName:}" failed. No retries permitted until 2026-07-21 21:50:05.248572731 +0000 UTC m=+83.011806973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/cf08e0a6-5fee-4612-9805-3c91164df632-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-tgl5d" (UID: "cf08e0a6-5fee-4612-9805-3c91164df632") : secret "monitoring-plugin-cert" not found Jul 21 21:50:04.758125 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.758095 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-456cn\" (UniqueName: \"kubernetes.io/projected/cf08e0a6-5fee-4612-9805-3c91164df632-kube-api-access-456cn\") pod \"monitoring-plugin-7b7b7bc8bf-tgl5d\" (UID: \"cf08e0a6-5fee-4612-9805-3c91164df632\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tgl5d" Jul 21 21:50:04.974022 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.973984 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-554d488d7f-77c9d"] Jul 21 21:50:04.977309 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.977286 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:04.980610 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.980575 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 21 21:50:04.980758 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.980708 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 21 21:50:04.980758 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.980736 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 21 21:50:04.980758 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.980737 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 21 21:50:04.980918 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.980736 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 21 21:50:04.980918 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.980844 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kxj2r\"" Jul 21 21:50:04.986435 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.986287 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 21 21:50:04.995362 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:04.995330 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-554d488d7f-77c9d"] Jul 21 21:50:05.152236 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.152150 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/94e868f0-a955-4150-a07a-0874828f0ced-federate-client-tls\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.152236 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.152222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/94e868f0-a955-4150-a07a-0874828f0ced-telemeter-client-tls\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.152468 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.152273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94e868f0-a955-4150-a07a-0874828f0ced-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.152468 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.152315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/94e868f0-a955-4150-a07a-0874828f0ced-secret-telemeter-client\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.152468 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.152344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxk9x\" (UniqueName: \"kubernetes.io/projected/94e868f0-a955-4150-a07a-0874828f0ced-kube-api-access-gxk9x\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.152468 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.152364 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94e868f0-a955-4150-a07a-0874828f0ced-telemeter-trusted-ca-bundle\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.152468 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.152458 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94e868f0-a955-4150-a07a-0874828f0ced-serving-certs-ca-bundle\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.152662 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.152493 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94e868f0-a955-4150-a07a-0874828f0ced-metrics-client-ca\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.253624 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.253582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94e868f0-a955-4150-a07a-0874828f0ced-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.254092 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.253635 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/94e868f0-a955-4150-a07a-0874828f0ced-secret-telemeter-client\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.254092 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.253772 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxk9x\" (UniqueName: \"kubernetes.io/projected/94e868f0-a955-4150-a07a-0874828f0ced-kube-api-access-gxk9x\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.254092 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.253823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94e868f0-a955-4150-a07a-0874828f0ced-telemeter-trusted-ca-bundle\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.254092 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.253863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94e868f0-a955-4150-a07a-0874828f0ced-serving-certs-ca-bundle\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.254092 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.253888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94e868f0-a955-4150-a07a-0874828f0ced-metrics-client-ca\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.254092 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.253975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cf08e0a6-5fee-4612-9805-3c91164df632-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-tgl5d\" (UID: \"cf08e0a6-5fee-4612-9805-3c91164df632\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tgl5d" Jul 21 21:50:05.254092 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.254000 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/94e868f0-a955-4150-a07a-0874828f0ced-federate-client-tls\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.254092 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.254053 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/94e868f0-a955-4150-a07a-0874828f0ced-telemeter-client-tls\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.254591 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.254566 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94e868f0-a955-4150-a07a-0874828f0ced-serving-certs-ca-bundle\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.255112 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.255086 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94e868f0-a955-4150-a07a-0874828f0ced-telemeter-trusted-ca-bundle\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.255403 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.255144 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94e868f0-a955-4150-a07a-0874828f0ced-metrics-client-ca\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.256759 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.256732 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94e868f0-a955-4150-a07a-0874828f0ced-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.256998 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.256969 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/94e868f0-a955-4150-a07a-0874828f0ced-secret-telemeter-client\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.257402 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.257353 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/94e868f0-a955-4150-a07a-0874828f0ced-telemeter-client-tls\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.257402 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.257359 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cf08e0a6-5fee-4612-9805-3c91164df632-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-tgl5d\" (UID: \"cf08e0a6-5fee-4612-9805-3c91164df632\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tgl5d" Jul 21 21:50:05.257862 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.257843 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/94e868f0-a955-4150-a07a-0874828f0ced-federate-client-tls\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.264324 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.264301 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxk9x\" (UniqueName: \"kubernetes.io/projected/94e868f0-a955-4150-a07a-0874828f0ced-kube-api-access-gxk9x\") pod \"telemeter-client-554d488d7f-77c9d\" (UID: \"94e868f0-a955-4150-a07a-0874828f0ced\") " pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.290024 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.289777 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" Jul 21 21:50:05.442901 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.442722 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tgl5d" Jul 21 21:50:05.528649 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.528620 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-554d488d7f-77c9d"] Jul 21 21:50:05.532356 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:50:05.532323 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e868f0_a955_4150_a07a_0874828f0ced.slice/crio-1ef971c2fd3a498ca051c518aa3646345c5c28eb5c06eb88a68bb082f085eae8 WatchSource:0}: Error finding container 1ef971c2fd3a498ca051c518aa3646345c5c28eb5c06eb88a68bb082f085eae8: Status 404 returned error can't find the container with id 1ef971c2fd3a498ca051c518aa3646345c5c28eb5c06eb88a68bb082f085eae8 Jul 21 21:50:05.555653 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.555279 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8645679d69-f7ndk"] Jul 21 21:50:05.643656 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.643346 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tgl5d"] Jul 21 21:50:05.969495 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.969393 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 21:50:05.973870 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.973845 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:05.977568 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.977343 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 21 21:50:05.977568 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.977397 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tbbmm\"" Jul 21 21:50:05.977568 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.977399 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 21 21:50:05.977568 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.977456 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 21 21:50:05.977568 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.977479 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 21 21:50:05.977568 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.977439 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 21 21:50:05.977884 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.977627 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ad9b629i8hids\"" Jul 21 21:50:05.978196 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.978082 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 21 21:50:05.978196 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.978130 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 21 21:50:05.978196 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.978140 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 21 21:50:05.978394 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.978327 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 21 21:50:05.978394 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.978345 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 21 21:50:05.978498 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.978434 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 21 21:50:05.978762 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.978733 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 21 21:50:05.981933 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.981782 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 21 21:50:05.984435 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.984417 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 21 21:50:05.989951 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:05.989859 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 21:50:06.065203 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065060 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.065203 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065129 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.065453 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065207 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.065453 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065244 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.065453 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065276 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-web-config\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.065453 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065346 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.065453 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065404 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.065453 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065436 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec671505-9cec-4be4-9db2-a51589875186-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.065756 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065471 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.065756 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065518 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dnw2\" (UniqueName: \"kubernetes.io/projected/ec671505-9cec-4be4-9db2-a51589875186-kube-api-access-7dnw2\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.065756 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065543 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.065756 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.065756 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065593 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.065756 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065635 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ec671505-9cec-4be4-9db2-a51589875186-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.065756 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065675 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec671505-9cec-4be4-9db2-a51589875186-config-out\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.065756 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065697 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.065756 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065722 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.065756 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065745 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-config\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.066279 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.066279 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.065794 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.170106 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.166771 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec671505-9cec-4be4-9db2-a51589875186-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.170106 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.166828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.170106 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.166876 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dnw2\" (UniqueName: \"kubernetes.io/projected/ec671505-9cec-4be4-9db2-a51589875186-kube-api-access-7dnw2\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.170106 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.166900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.170106 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.166923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.170106 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.166947 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.170106 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.166984 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ec671505-9cec-4be4-9db2-a51589875186-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.170106 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.167108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec671505-9cec-4be4-9db2-a51589875186-config-out\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.170106 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.167130 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.170106 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.167153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.170106 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.167178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-config\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.170106 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.167200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.170106 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.167223 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.170106 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.167273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.170106 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.167326 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.170106 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.167364 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.173336 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.171947 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.173336 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.171950 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.173336 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.172827 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.176593 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.173339 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ec671505-9cec-4be4-9db2-a51589875186-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.176593 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.173418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.176593 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.173468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-web-config\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.176593 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.173512 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.176593 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.173545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.176593 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.173623 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.176593 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.174134 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.176593 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.174665 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.176593 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.174882 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec671505-9cec-4be4-9db2-a51589875186-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.176593 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.175783 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.177410 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.177050 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.177410 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.177340 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ec671505-9cec-4be4-9db2-a51589875186-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.177972 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.177949 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-web-config\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.178048 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.178007 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.178647 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.178566 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.179733 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.179689 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.181853 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.180259 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.181853 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.181483 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-config\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.181853 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.181810 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec671505-9cec-4be4-9db2-a51589875186-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.182420 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.182394 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec671505-9cec-4be4-9db2-a51589875186-config-out\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.183106 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.183063 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dnw2\" (UniqueName: \"kubernetes.io/projected/ec671505-9cec-4be4-9db2-a51589875186-kube-api-access-7dnw2\") pod \"prometheus-k8s-0\" (UID: \"ec671505-9cec-4be4-9db2-a51589875186\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.285885 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.285626 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:06.287354 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.287270 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tgl5d" event={"ID":"cf08e0a6-5fee-4612-9805-3c91164df632","Type":"ContainerStarted","Data":"77a50e6d0584b12fff3d6a76898221261846a1df054e2561676d0485b30fc7cb"} Jul 21 21:50:06.291629 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.291435 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-644856656-r8znz" event={"ID":"860ea5df-15b1-473d-a82b-fb8b89136cee","Type":"ContainerStarted","Data":"20b8268b36d8711c65e1b736e28df1d3157c0c864e3249424de80dd303928ae7"} Jul 21 21:50:06.291629 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.291520 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-644856656-r8znz" event={"ID":"860ea5df-15b1-473d-a82b-fb8b89136cee","Type":"ContainerStarted","Data":"8e59ab5ffd043e1fd03cb4f28170b5dfd5ed56ffb6bff72a95843309034454e6"} Jul 21 21:50:06.291629 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.291564 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-644856656-r8znz" event={"ID":"860ea5df-15b1-473d-a82b-fb8b89136cee","Type":"ContainerStarted","Data":"72b4078f729f270f86e101849353270455c202535e08117170fb05436b176179"} Jul 21 21:50:06.292884 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.292861 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" event={"ID":"94e868f0-a955-4150-a07a-0874828f0ced","Type":"ContainerStarted","Data":"1ef971c2fd3a498ca051c518aa3646345c5c28eb5c06eb88a68bb082f085eae8"} Jul 21 21:50:06.298918 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.298684 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91","Type":"ContainerStarted","Data":"90eb315cbd385729cbc685e510eb2a24d4634a089fbfff945b4e46d14ea3bef2"} Jul 21 21:50:06.298918 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.298751 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91","Type":"ContainerStarted","Data":"47b4ce38f7fa926514340f549b072f214f29653e68fee722a73a4cf396d20188"} Jul 21 21:50:06.298918 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.298789 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91","Type":"ContainerStarted","Data":"752e95582eb645bb798c7bebf76a2b50cb3e5ab6468b4b1602304dc62486d1ad"} Jul 21 21:50:06.298918 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.298825 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91","Type":"ContainerStarted","Data":"b299c9f113a9733ed0b194a15e08624aa9067f8811e6e94c6cac0a045603b702"} Jul 21 21:50:06.298918 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.298864 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91","Type":"ContainerStarted","Data":"1f269c17a92aacb1869bd3f3517424aa8e8716465ac86ec6793f3911d23cd1cd"} Jul 21 21:50:06.300671 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.300505 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" event={"ID":"19851a0f-4eac-44f8-b73c-dfbc747236cf","Type":"ContainerStarted","Data":"52e5982cdd569ac177ddfa14f75c3720bcf63a1600079dbc23ea7b1f2d911cd2"} Jul 21 21:50:06.511711 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:06.511595 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 21:50:06.671015 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:50:06.670937 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec671505_9cec_4be4_9db2_a51589875186.slice/crio-e732862006a20ad4b3b760437a6bb07bd3c93d89c7843aa99912b88b0e854ca5 WatchSource:0}: Error finding container e732862006a20ad4b3b760437a6bb07bd3c93d89c7843aa99912b88b0e854ca5: Status 404 returned error can't find the container with id e732862006a20ad4b3b760437a6bb07bd3c93d89c7843aa99912b88b0e854ca5 Jul 21 21:50:07.310503 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:07.310460 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-644856656-r8znz" event={"ID":"860ea5df-15b1-473d-a82b-fb8b89136cee","Type":"ContainerStarted","Data":"f9fd0b49f2f5d2f97f49ac146c36e199807f0beca64775bcd62b03211eca1aa7"} Jul 21 21:50:07.310982 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:07.310558 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-644856656-r8znz" event={"ID":"860ea5df-15b1-473d-a82b-fb8b89136cee","Type":"ContainerStarted","Data":"2ec34c838a3389d5728181d04daf0a0219fa250fdbbbdf1369cdcbdf7858578f"} Jul 21 21:50:07.314833 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:07.314799 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f8ecf16-5c8e-48e1-bebc-cc747db6dd91","Type":"ContainerStarted","Data":"7220e66840abc8b734f448b584ccf4188e2d7d46523835d4e9654ab6a25a33d0"} Jul 21 21:50:07.317004 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:07.316967 2580 generic.go:356] "Generic (PLEG): container finished" podID="ec671505-9cec-4be4-9db2-a51589875186" containerID="5d2695af34f16a02e0445189498dadb66ae4076fbd1e5a9478a6c413d667b537" exitCode=0 Jul 21 21:50:07.317139 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:07.317018 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec671505-9cec-4be4-9db2-a51589875186","Type":"ContainerDied","Data":"5d2695af34f16a02e0445189498dadb66ae4076fbd1e5a9478a6c413d667b537"} Jul 21 21:50:07.317139 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:07.317099 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec671505-9cec-4be4-9db2-a51589875186","Type":"ContainerStarted","Data":"e732862006a20ad4b3b760437a6bb07bd3c93d89c7843aa99912b88b0e854ca5"} Jul 21 21:50:07.345775 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:07.345679 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.5555180699999998 podStartE2EDuration="7.345658512s" totalImagesPullingTime="4.766270273s" totalInitContainerRuntime="23.870169ms" isStatefulPod=false podCreationTimestamp="2026-07-21 21:50:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:50:07.344927093 +0000 UTC m=+85.108161355" watchObservedRunningTime="2026-07-21 21:50:07.345658512 +0000 UTC m=+85.108892762" Jul 21 21:50:08.325347 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:08.325310 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tgl5d" event={"ID":"cf08e0a6-5fee-4612-9805-3c91164df632","Type":"ContainerStarted","Data":"b6fa9de9985d94a1c8cf2dcc1a7fca432c352c78e124afe73e9e9a50c7b2ed96"} Jul 21 21:50:08.326651 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:08.326591 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tgl5d" Jul 21 21:50:08.335008 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:08.334890 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-644856656-r8znz" event={"ID":"860ea5df-15b1-473d-a82b-fb8b89136cee","Type":"ContainerStarted","Data":"2f553e38c0504768e63ceb8db5c6813b8f422c40d93c9774b44f284e0196287c"} Jul 21 21:50:08.335472 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:08.335447 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:08.336032 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:08.335987 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tgl5d" Jul 21 21:50:08.338643 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:08.338618 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" event={"ID":"94e868f0-a955-4150-a07a-0874828f0ced","Type":"ContainerStarted","Data":"2f4e4b7a88d97163a60bb87437ec3a123201ba1772143340e526013dba87b1a6"} Jul 21 21:50:08.340922 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:08.340859 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" event={"ID":"19851a0f-4eac-44f8-b73c-dfbc747236cf","Type":"ContainerStarted","Data":"f0ff3e9045fc22946fa0ffa5e05a796a8d93beb3271b7864db04c699ea9b5270"} Jul 21 21:50:08.346119 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:08.346073 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tgl5d" podStartSLOduration=1.8152618029999998 podStartE2EDuration="4.346053128s" totalImagesPullingTime="2.530791325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:50:08.344321989 +0000 UTC m=+86.107556249" watchObservedRunningTime="2026-07-21 21:50:08.346053128 +0000 UTC m=+86.109287371" Jul 21 21:50:08.369095 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:08.368040 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-644856656-r8znz" podStartSLOduration=3.741872007 podStartE2EDuration="7.368010434s" totalImagesPullingTime="3.626138427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:50:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:50:08.366872471 +0000 UTC m=+86.130106751" watchObservedRunningTime="2026-07-21 21:50:08.368010434 +0000 UTC m=+86.131244695" Jul 21 21:50:08.386331 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:08.386156 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" podStartSLOduration=1.759975563 podStartE2EDuration="4.386142091s" totalImagesPullingTime="2.626166528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:50:08.384130451 +0000 UTC m=+86.147364714" watchObservedRunningTime="2026-07-21 21:50:08.386142091 +0000 UTC m=+86.149376353" Jul 21 21:50:09.348115 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:09.348074 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" event={"ID":"94e868f0-a955-4150-a07a-0874828f0ced","Type":"ContainerStarted","Data":"d501a69ab5990081b2b66638c8843df33d58c11fcf535443d345b744932ceb4b"} Jul 21 21:50:09.348605 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:09.348147 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" event={"ID":"94e868f0-a955-4150-a07a-0874828f0ced","Type":"ContainerStarted","Data":"270159c1f36c6ac861522f5eb450dd85748570e7eeadbe5eeac415b98b74fa72"} Jul 21 21:50:09.372435 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:09.372340 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-554d488d7f-77c9d" podStartSLOduration=2.723098766 podStartE2EDuration="5.37231862s" totalImagesPullingTime="2.649219854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:50:09.370558517 +0000 UTC m=+87.133792779" watchObservedRunningTime="2026-07-21 21:50:09.37231862 +0000 UTC m=+87.135552885" Jul 21 21:50:11.363401 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:11.363346 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec671505-9cec-4be4-9db2-a51589875186","Type":"ContainerStarted","Data":"442739ca9941ccf9992417bfa3b5e5315cd0a69edf8e541348aaed5868ff93db"} Jul 21 21:50:11.363823 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:11.363511 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec671505-9cec-4be4-9db2-a51589875186","Type":"ContainerStarted","Data":"6d2c68d1c8a50be5981a831e95fc475253ee9d93929c29ac3ef163ce44d58bb0"} Jul 21 21:50:11.363823 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:11.363560 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec671505-9cec-4be4-9db2-a51589875186","Type":"ContainerStarted","Data":"9a4e75e158722d2fb274b26afb7b1a675e959ee62435714554b35727dab0dd7f"} Jul 21 21:50:11.363823 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:11.363598 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec671505-9cec-4be4-9db2-a51589875186","Type":"ContainerStarted","Data":"6b77faa6f3028c60bc8c0db3766a480935421286135392205f31c6eb8ccaadae"} Jul 21 21:50:12.370719 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:12.370686 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec671505-9cec-4be4-9db2-a51589875186","Type":"ContainerStarted","Data":"74fe46c7c2191dc8db31e72c5c89cb6b462b7c1472ffca723366236e5ab51b5c"} Jul 21 21:50:12.371114 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:12.370750 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec671505-9cec-4be4-9db2-a51589875186","Type":"ContainerStarted","Data":"c855513f6279e02bb26f476f4c5f68be1ef7adfb1e7675833c6c925be7129ce6"} Jul 21 21:50:12.404147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:12.404087 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.66357006 podStartE2EDuration="7.404068845s" totalImagesPullingTime="3.714227298s" totalInitContainerRuntime="26.271487ms" isStatefulPod=false podCreationTimestamp="2026-07-21 21:50:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:50:12.401625477 +0000 UTC m=+90.164859737" watchObservedRunningTime="2026-07-21 21:50:12.404068845 +0000 UTC m=+90.167303148" Jul 21 21:50:14.354620 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:14.354594 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-644856656-r8znz" Jul 21 21:50:16.287469 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:16.287429 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:50:20.997153 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:20.997126 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54f5d79568-5vmjl"] Jul 21 21:50:24.509435 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:24.509400 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:24.509831 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:24.509498 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:29.225369 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:29.225336 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zrk2g" Jul 21 21:50:44.514347 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:44.514317 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:44.518016 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:44.517993 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-8645679d69-f7ndk" Jul 21 21:50:46.025340 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.025261 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54f5d79568-5vmjl" podUID="6c341c9f-8559-4335-8624-82f38b0cdbeb" containerName="console" containerID="cri-o://2a3610196e4a1e9c3a9dd8cfeb0a9ebe41e28f2e6a2cc8df5ee07470120521e7" gracePeriod=15 Jul 21 21:50:46.276814 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.276760 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:50:46.448648 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.448618 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-trusted-ca-bundle\") pod \"6c341c9f-8559-4335-8624-82f38b0cdbeb\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " Jul 21 21:50:46.448785 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.448661 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbrsr\" (UniqueName: \"kubernetes.io/projected/6c341c9f-8559-4335-8624-82f38b0cdbeb-kube-api-access-jbrsr\") pod \"6c341c9f-8559-4335-8624-82f38b0cdbeb\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " Jul 21 21:50:46.448785 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.448703 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-service-ca\") pod \"6c341c9f-8559-4335-8624-82f38b0cdbeb\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " Jul 21 21:50:46.448908 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.448809 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c341c9f-8559-4335-8624-82f38b0cdbeb-console-serving-cert\") pod \"6c341c9f-8559-4335-8624-82f38b0cdbeb\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " Jul 21 21:50:46.448908 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.448877 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c341c9f-8559-4335-8624-82f38b0cdbeb-console-oauth-config\") pod \"6c341c9f-8559-4335-8624-82f38b0cdbeb\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " Jul 21 21:50:46.449056 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.448913 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-oauth-serving-cert\") pod \"6c341c9f-8559-4335-8624-82f38b0cdbeb\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " Jul 21 21:50:46.449056 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.448941 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-console-config\") pod \"6c341c9f-8559-4335-8624-82f38b0cdbeb\" (UID: \"6c341c9f-8559-4335-8624-82f38b0cdbeb\") " Jul 21 21:50:46.449156 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.449043 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6c341c9f-8559-4335-8624-82f38b0cdbeb" (UID: "6c341c9f-8559-4335-8624-82f38b0cdbeb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:50:46.449233 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.449205 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6c341c9f-8559-4335-8624-82f38b0cdbeb" (UID: "6c341c9f-8559-4335-8624-82f38b0cdbeb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:50:46.449318 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.449296 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-trusted-ca-bundle\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:50:46.449393 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.449324 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-oauth-serving-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:50:46.449393 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.449336 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-console-config" (OuterVolumeSpecName: "console-config") pod "6c341c9f-8559-4335-8624-82f38b0cdbeb" (UID: "6c341c9f-8559-4335-8624-82f38b0cdbeb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:50:46.449663 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.449641 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-service-ca" (OuterVolumeSpecName: "service-ca") pod "6c341c9f-8559-4335-8624-82f38b0cdbeb" (UID: "6c341c9f-8559-4335-8624-82f38b0cdbeb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:50:46.451251 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.451227 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c341c9f-8559-4335-8624-82f38b0cdbeb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6c341c9f-8559-4335-8624-82f38b0cdbeb" (UID: "6c341c9f-8559-4335-8624-82f38b0cdbeb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:50:46.451517 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.451489 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c341c9f-8559-4335-8624-82f38b0cdbeb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6c341c9f-8559-4335-8624-82f38b0cdbeb" (UID: "6c341c9f-8559-4335-8624-82f38b0cdbeb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:50:46.451664 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.451645 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c341c9f-8559-4335-8624-82f38b0cdbeb-kube-api-access-jbrsr" (OuterVolumeSpecName: "kube-api-access-jbrsr") pod "6c341c9f-8559-4335-8624-82f38b0cdbeb" (UID: "6c341c9f-8559-4335-8624-82f38b0cdbeb"). InnerVolumeSpecName "kube-api-access-jbrsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:50:46.509323 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.509286 2580 generic.go:356] "Generic (PLEG): container finished" podID="6c341c9f-8559-4335-8624-82f38b0cdbeb" containerID="2a3610196e4a1e9c3a9dd8cfeb0a9ebe41e28f2e6a2cc8df5ee07470120521e7" exitCode=0 Jul 21 21:50:46.509469 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.509343 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f5d79568-5vmjl" Jul 21 21:50:46.509469 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.509390 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f5d79568-5vmjl" event={"ID":"6c341c9f-8559-4335-8624-82f38b0cdbeb","Type":"ContainerDied","Data":"2a3610196e4a1e9c3a9dd8cfeb0a9ebe41e28f2e6a2cc8df5ee07470120521e7"} Jul 21 21:50:46.509469 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.509449 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f5d79568-5vmjl" event={"ID":"6c341c9f-8559-4335-8624-82f38b0cdbeb","Type":"ContainerDied","Data":"b852a63d4004de97c19c6b6f1c56546da9527108811f2ab0caa2ed3f3bad0d42"} Jul 21 21:50:46.509621 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.509476 2580 scope.go:117] "RemoveContainer" containerID="2a3610196e4a1e9c3a9dd8cfeb0a9ebe41e28f2e6a2cc8df5ee07470120521e7" Jul 21 21:50:46.518982 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.518964 2580 scope.go:117] "RemoveContainer" containerID="2a3610196e4a1e9c3a9dd8cfeb0a9ebe41e28f2e6a2cc8df5ee07470120521e7" Jul 21 21:50:46.519223 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:50:46.519204 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a3610196e4a1e9c3a9dd8cfeb0a9ebe41e28f2e6a2cc8df5ee07470120521e7\": container with ID starting with 2a3610196e4a1e9c3a9dd8cfeb0a9ebe41e28f2e6a2cc8df5ee07470120521e7 not found: ID does not exist" containerID="2a3610196e4a1e9c3a9dd8cfeb0a9ebe41e28f2e6a2cc8df5ee07470120521e7" Jul 21 21:50:46.519290 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.519231 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a3610196e4a1e9c3a9dd8cfeb0a9ebe41e28f2e6a2cc8df5ee07470120521e7"} err="failed to get container status \"2a3610196e4a1e9c3a9dd8cfeb0a9ebe41e28f2e6a2cc8df5ee07470120521e7\": rpc error: code = NotFound desc = could not find container \"2a3610196e4a1e9c3a9dd8cfeb0a9ebe41e28f2e6a2cc8df5ee07470120521e7\": container with ID starting with 2a3610196e4a1e9c3a9dd8cfeb0a9ebe41e28f2e6a2cc8df5ee07470120521e7 not found: ID does not exist" Jul 21 21:50:46.532307 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.532255 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54f5d79568-5vmjl"] Jul 21 21:50:46.536416 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.536396 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54f5d79568-5vmjl"] Jul 21 21:50:46.550088 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.550067 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c341c9f-8559-4335-8624-82f38b0cdbeb-console-oauth-config\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:50:46.550193 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.550090 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-console-config\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:50:46.550193 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.550105 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbrsr\" (UniqueName: \"kubernetes.io/projected/6c341c9f-8559-4335-8624-82f38b0cdbeb-kube-api-access-jbrsr\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:50:46.550193 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.550116 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c341c9f-8559-4335-8624-82f38b0cdbeb-service-ca\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:50:46.550193 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.550130 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c341c9f-8559-4335-8624-82f38b0cdbeb-console-serving-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:50:46.778558 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:46.778527 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c341c9f-8559-4335-8624-82f38b0cdbeb" path="/var/lib/kubelet/pods/6c341c9f-8559-4335-8624-82f38b0cdbeb/volumes" Jul 21 21:50:47.203949 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:47.203910 2580 patch_prober.go:28] interesting pod/console-54f5d79568-5vmjl container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.10:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 21 21:50:47.204303 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:50:47.203991 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-54f5d79568-5vmjl" podUID="6c341c9f-8559-4335-8624-82f38b0cdbeb" containerName="console" probeResult="failure" output="Get \"https://10.134.0.10:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 21 21:51:06.287612 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:51:06.287552 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:51:06.308154 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:51:06.308134 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:51:06.607738 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:51:06.607663 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:53:14.133548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.133518 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9kh97"] Jul 21 21:53:14.134132 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.133857 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c341c9f-8559-4335-8624-82f38b0cdbeb" containerName="console" Jul 21 21:53:14.134132 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.133866 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c341c9f-8559-4335-8624-82f38b0cdbeb" containerName="console" Jul 21 21:53:14.134132 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.133923 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c341c9f-8559-4335-8624-82f38b0cdbeb" containerName="console" Jul 21 21:53:14.136754 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.136739 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9kh97" Jul 21 21:53:14.139463 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.139433 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 21 21:53:14.140401 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.140387 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:53:14.140401 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.140396 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-974l9\"" Jul 21 21:53:14.151398 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.151338 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9kh97"] Jul 21 21:53:14.254666 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.254643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db2829c9-411b-4d5a-af2d-0c3c753ff36a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9kh97\" (UID: \"db2829c9-411b-4d5a-af2d-0c3c753ff36a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9kh97" Jul 21 21:53:14.254791 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.254697 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmtgl\" (UniqueName: \"kubernetes.io/projected/db2829c9-411b-4d5a-af2d-0c3c753ff36a-kube-api-access-cmtgl\") pod \"openshift-lws-operator-fd8ccff4c-9kh97\" (UID: \"db2829c9-411b-4d5a-af2d-0c3c753ff36a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9kh97" Jul 21 21:53:14.306387 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.306351 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gz5mm"] Jul 21 21:53:14.309931 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.309915 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-gz5mm" Jul 21 21:53:14.312669 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.312651 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 21 21:53:14.312669 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.312666 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 21 21:53:14.312799 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.312666 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-nx7m5\"" Jul 21 21:53:14.319931 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.319911 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gz5mm"] Jul 21 21:53:14.355638 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.355615 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmtgl\" (UniqueName: \"kubernetes.io/projected/db2829c9-411b-4d5a-af2d-0c3c753ff36a-kube-api-access-cmtgl\") pod \"openshift-lws-operator-fd8ccff4c-9kh97\" (UID: \"db2829c9-411b-4d5a-af2d-0c3c753ff36a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9kh97" Jul 21 21:53:14.355730 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.355654 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hw6x\" (UniqueName: \"kubernetes.io/projected/01f42372-ba02-4f33-ac7e-6b2580597dc8-kube-api-access-7hw6x\") pod \"cert-manager-55b4d954d5-gz5mm\" (UID: \"01f42372-ba02-4f33-ac7e-6b2580597dc8\") " pod="cert-manager/cert-manager-55b4d954d5-gz5mm" Jul 21 21:53:14.355730 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.355688 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db2829c9-411b-4d5a-af2d-0c3c753ff36a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9kh97\" (UID: \"db2829c9-411b-4d5a-af2d-0c3c753ff36a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9kh97" Jul 21 21:53:14.355805 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.355732 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01f42372-ba02-4f33-ac7e-6b2580597dc8-bound-sa-token\") pod \"cert-manager-55b4d954d5-gz5mm\" (UID: \"01f42372-ba02-4f33-ac7e-6b2580597dc8\") " pod="cert-manager/cert-manager-55b4d954d5-gz5mm" Jul 21 21:53:14.356035 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.356020 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db2829c9-411b-4d5a-af2d-0c3c753ff36a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9kh97\" (UID: \"db2829c9-411b-4d5a-af2d-0c3c753ff36a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9kh97" Jul 21 21:53:14.365029 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.365010 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmtgl\" (UniqueName: \"kubernetes.io/projected/db2829c9-411b-4d5a-af2d-0c3c753ff36a-kube-api-access-cmtgl\") pod \"openshift-lws-operator-fd8ccff4c-9kh97\" (UID: \"db2829c9-411b-4d5a-af2d-0c3c753ff36a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9kh97" Jul 21 21:53:14.456217 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.456165 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01f42372-ba02-4f33-ac7e-6b2580597dc8-bound-sa-token\") pod \"cert-manager-55b4d954d5-gz5mm\" (UID: \"01f42372-ba02-4f33-ac7e-6b2580597dc8\") " pod="cert-manager/cert-manager-55b4d954d5-gz5mm" Jul 21 21:53:14.456217 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.456204 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hw6x\" (UniqueName: \"kubernetes.io/projected/01f42372-ba02-4f33-ac7e-6b2580597dc8-kube-api-access-7hw6x\") pod \"cert-manager-55b4d954d5-gz5mm\" (UID: \"01f42372-ba02-4f33-ac7e-6b2580597dc8\") " pod="cert-manager/cert-manager-55b4d954d5-gz5mm" Jul 21 21:53:14.461150 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.461135 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9kh97" Jul 21 21:53:14.464856 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.464838 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hw6x\" (UniqueName: \"kubernetes.io/projected/01f42372-ba02-4f33-ac7e-6b2580597dc8-kube-api-access-7hw6x\") pod \"cert-manager-55b4d954d5-gz5mm\" (UID: \"01f42372-ba02-4f33-ac7e-6b2580597dc8\") " pod="cert-manager/cert-manager-55b4d954d5-gz5mm" Jul 21 21:53:14.464945 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.464865 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01f42372-ba02-4f33-ac7e-6b2580597dc8-bound-sa-token\") pod \"cert-manager-55b4d954d5-gz5mm\" (UID: \"01f42372-ba02-4f33-ac7e-6b2580597dc8\") " pod="cert-manager/cert-manager-55b4d954d5-gz5mm" Jul 21 21:53:14.588242 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.588221 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9kh97"] Jul 21 21:53:14.590057 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:53:14.590033 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb2829c9_411b_4d5a_af2d_0c3c753ff36a.slice/crio-3d11c0ef54c503bc70adf786a3acf06b83d8a7f14ec6b6847a2e2e73748724ba WatchSource:0}: Error finding container 3d11c0ef54c503bc70adf786a3acf06b83d8a7f14ec6b6847a2e2e73748724ba: Status 404 returned error can't find the container with id 3d11c0ef54c503bc70adf786a3acf06b83d8a7f14ec6b6847a2e2e73748724ba Jul 21 21:53:14.618777 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.618755 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-gz5mm" Jul 21 21:53:14.745944 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:14.745899 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gz5mm"] Jul 21 21:53:14.748004 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:53:14.747971 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01f42372_ba02_4f33_ac7e_6b2580597dc8.slice/crio-36767530a77fd9e146bdfe72c55941e09d6c22c8ba31f4c42ce15e951c201c49 WatchSource:0}: Error finding container 36767530a77fd9e146bdfe72c55941e09d6c22c8ba31f4c42ce15e951c201c49: Status 404 returned error can't find the container with id 36767530a77fd9e146bdfe72c55941e09d6c22c8ba31f4c42ce15e951c201c49 Jul 21 21:53:15.090275 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:15.090170 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-gz5mm" event={"ID":"01f42372-ba02-4f33-ac7e-6b2580597dc8","Type":"ContainerStarted","Data":"36767530a77fd9e146bdfe72c55941e09d6c22c8ba31f4c42ce15e951c201c49"} Jul 21 21:53:15.091749 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:15.091725 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9kh97" event={"ID":"db2829c9-411b-4d5a-af2d-0c3c753ff36a","Type":"ContainerStarted","Data":"3d11c0ef54c503bc70adf786a3acf06b83d8a7f14ec6b6847a2e2e73748724ba"} Jul 21 21:53:19.117586 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:19.117550 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-gz5mm" event={"ID":"01f42372-ba02-4f33-ac7e-6b2580597dc8","Type":"ContainerStarted","Data":"ac8b9239cf2a13aa02a83af45fcfeec8e22dc8c28a7bb3a6b0cfc79de1705f31"} Jul 21 21:53:19.119294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:19.119274 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9kh97" event={"ID":"db2829c9-411b-4d5a-af2d-0c3c753ff36a","Type":"ContainerStarted","Data":"22030aba5d39a515a19bc9eff729f8f75c674aafe67b2d0bc1b06e2810c69754"} Jul 21 21:53:19.136630 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:19.136559 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-gz5mm" podStartSLOduration=1.003139542 podStartE2EDuration="5.136545539s" totalImagesPullingTime="4.133405997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:53:19.135103581 +0000 UTC m=+276.898337842" watchObservedRunningTime="2026-07-21 21:53:19.136545539 +0000 UTC m=+276.899779800" Jul 21 21:53:19.151859 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:19.151816 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9kh97" podStartSLOduration=0.86575183 podStartE2EDuration="5.151797619s" totalImagesPullingTime="4.286045789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:53:19.151596752 +0000 UTC m=+276.914831014" watchObservedRunningTime="2026-07-21 21:53:19.151797619 +0000 UTC m=+276.915031871" Jul 21 21:53:38.731559 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.731523 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p"] Jul 21 21:53:38.738019 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.738002 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:53:38.743122 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.743103 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fdpkb\"" Jul 21 21:53:38.743645 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.743623 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 21:53:38.744046 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.744027 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 21:53:38.744268 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.744249 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 21:53:38.744400 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.744310 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 21:53:38.835738 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.835710 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p"] Jul 21 21:53:38.873123 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.873088 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84aa0565-abdc-41bc-a77b-29f4f022854e-tmp\") pod \"opendatahub-operator-controller-manager-54667d85dc-lwv8p\" (UID: \"84aa0565-abdc-41bc-a77b-29f4f022854e\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:53:38.873280 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.873131 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5czdp\" (UniqueName: \"kubernetes.io/projected/84aa0565-abdc-41bc-a77b-29f4f022854e-kube-api-access-5czdp\") pod \"opendatahub-operator-controller-manager-54667d85dc-lwv8p\" (UID: \"84aa0565-abdc-41bc-a77b-29f4f022854e\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:53:38.873280 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.873150 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/84aa0565-abdc-41bc-a77b-29f4f022854e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54667d85dc-lwv8p\" (UID: \"84aa0565-abdc-41bc-a77b-29f4f022854e\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:53:38.873280 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.873198 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/84aa0565-abdc-41bc-a77b-29f4f022854e-webhook-cert\") pod \"opendatahub-operator-controller-manager-54667d85dc-lwv8p\" (UID: \"84aa0565-abdc-41bc-a77b-29f4f022854e\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:53:38.873280 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.873219 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/84aa0565-abdc-41bc-a77b-29f4f022854e-manifests\") pod \"opendatahub-operator-controller-manager-54667d85dc-lwv8p\" (UID: \"84aa0565-abdc-41bc-a77b-29f4f022854e\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:53:38.974574 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.974537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/84aa0565-abdc-41bc-a77b-29f4f022854e-manifests\") pod \"opendatahub-operator-controller-manager-54667d85dc-lwv8p\" (UID: \"84aa0565-abdc-41bc-a77b-29f4f022854e\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:53:38.974779 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.974608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84aa0565-abdc-41bc-a77b-29f4f022854e-tmp\") pod \"opendatahub-operator-controller-manager-54667d85dc-lwv8p\" (UID: \"84aa0565-abdc-41bc-a77b-29f4f022854e\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:53:38.974779 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.974633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5czdp\" (UniqueName: \"kubernetes.io/projected/84aa0565-abdc-41bc-a77b-29f4f022854e-kube-api-access-5czdp\") pod \"opendatahub-operator-controller-manager-54667d85dc-lwv8p\" (UID: \"84aa0565-abdc-41bc-a77b-29f4f022854e\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:53:38.974779 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.974653 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/84aa0565-abdc-41bc-a77b-29f4f022854e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54667d85dc-lwv8p\" (UID: \"84aa0565-abdc-41bc-a77b-29f4f022854e\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:53:38.974779 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.974678 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/84aa0565-abdc-41bc-a77b-29f4f022854e-webhook-cert\") pod \"opendatahub-operator-controller-manager-54667d85dc-lwv8p\" (UID: \"84aa0565-abdc-41bc-a77b-29f4f022854e\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:53:38.975031 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.975008 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/84aa0565-abdc-41bc-a77b-29f4f022854e-manifests\") pod \"opendatahub-operator-controller-manager-54667d85dc-lwv8p\" (UID: \"84aa0565-abdc-41bc-a77b-29f4f022854e\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:53:38.977145 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.977112 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84aa0565-abdc-41bc-a77b-29f4f022854e-tmp\") pod \"opendatahub-operator-controller-manager-54667d85dc-lwv8p\" (UID: \"84aa0565-abdc-41bc-a77b-29f4f022854e\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:53:38.977263 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.977200 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/84aa0565-abdc-41bc-a77b-29f4f022854e-webhook-cert\") pod \"opendatahub-operator-controller-manager-54667d85dc-lwv8p\" (UID: \"84aa0565-abdc-41bc-a77b-29f4f022854e\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:53:38.977263 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.977232 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/84aa0565-abdc-41bc-a77b-29f4f022854e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54667d85dc-lwv8p\" (UID: \"84aa0565-abdc-41bc-a77b-29f4f022854e\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:53:38.985363 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:38.985338 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5czdp\" (UniqueName: \"kubernetes.io/projected/84aa0565-abdc-41bc-a77b-29f4f022854e-kube-api-access-5czdp\") pod \"opendatahub-operator-controller-manager-54667d85dc-lwv8p\" (UID: \"84aa0565-abdc-41bc-a77b-29f4f022854e\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:53:39.049036 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:39.048999 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:53:39.196307 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:39.196169 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p"] Jul 21 21:53:39.198585 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:53:39.198555 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84aa0565_abdc_41bc_a77b_29f4f022854e.slice/crio-0a772b3ed8af738d00935c904a9ede5f5badfa92356b3bfc51c870044cfb7372 WatchSource:0}: Error finding container 0a772b3ed8af738d00935c904a9ede5f5badfa92356b3bfc51c870044cfb7372: Status 404 returned error can't find the container with id 0a772b3ed8af738d00935c904a9ede5f5badfa92356b3bfc51c870044cfb7372 Jul 21 21:53:39.204582 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:39.204549 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" event={"ID":"84aa0565-abdc-41bc-a77b-29f4f022854e","Type":"ContainerStarted","Data":"0a772b3ed8af738d00935c904a9ede5f5badfa92356b3bfc51c870044cfb7372"} Jul 21 21:53:42.611012 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:42.610988 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 21:53:43.227964 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:43.227918 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" event={"ID":"84aa0565-abdc-41bc-a77b-29f4f022854e","Type":"ContainerStarted","Data":"5498ecfaa4879c41c62127c530e42850d69a3169c7247df0c3521677e55e4039"} Jul 21 21:53:44.233510 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:44.233420 2580 generic.go:356] "Generic (PLEG): container finished" podID="84aa0565-abdc-41bc-a77b-29f4f022854e" containerID="5498ecfaa4879c41c62127c530e42850d69a3169c7247df0c3521677e55e4039" exitCode=0 Jul 21 21:53:44.233929 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:44.233501 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" event={"ID":"84aa0565-abdc-41bc-a77b-29f4f022854e","Type":"ContainerDied","Data":"5498ecfaa4879c41c62127c530e42850d69a3169c7247df0c3521677e55e4039"} Jul 21 21:53:44.233978 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:44.233964 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:53:45.239763 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:45.239725 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" event={"ID":"84aa0565-abdc-41bc-a77b-29f4f022854e","Type":"ContainerStarted","Data":"73d8c2834050bc8ee0dfa7bf694fbe33402327bf426c77570b7d94813cbf8b7e"} Jul 21 21:53:45.240192 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:45.239844 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:53:45.263681 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:45.263635 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" podStartSLOduration=2.14712422 podStartE2EDuration="7.263620427s" totalImagesPullingTime="4.11281219s" totalInitContainerRuntime="1.003684017s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:53:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:53:45.262136872 +0000 UTC m=+303.025371132" watchObservedRunningTime="2026-07-21 21:53:45.263620427 +0000 UTC m=+303.026854687" Jul 21 21:53:56.245612 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:53:56.245586 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-lwv8p" Jul 21 21:54:07.712394 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.712329 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7"] Jul 21 21:54:07.715670 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.715648 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.719846 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.719827 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 21 21:54:07.720585 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.720571 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 21:54:07.720650 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.720587 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-kjdqv\"" Jul 21 21:54:07.720865 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.720851 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 21:54:07.736888 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.736864 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7"] Jul 21 21:54:07.817280 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.817241 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.817280 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.817281 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.817524 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.817298 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.817524 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.817343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.817524 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.817406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.817524 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.817454 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.817524 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.817517 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.817689 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.817534 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.817689 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.817601 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5mjq\" (UniqueName: \"kubernetes.io/projected/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-kube-api-access-x5mjq\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.918620 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.918583 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5mjq\" (UniqueName: \"kubernetes.io/projected/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-kube-api-access-x5mjq\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.918815 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.918626 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.918815 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.918653 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.918815 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.918676 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.918815 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.918710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.918815 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.918742 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.918815 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.918805 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.919117 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.918878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.919117 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.918909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.919216 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.919120 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.919348 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.919324 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.919348 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.919338 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.919493 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.919394 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.919539 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.919523 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.921309 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.921283 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.921561 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.921542 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.938975 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.938948 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:07.940121 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:07.940097 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5mjq\" (UniqueName: \"kubernetes.io/projected/d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b-kube-api-access-x5mjq\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vpjh7\" (UID: \"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:08.026184 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.026160 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:08.183553 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.183528 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7"] Jul 21 21:54:08.186047 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:54:08.186023 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd557bb12_34d9_4b8b_bd9f_e8b1cf152e1b.slice/crio-14f87508b96abe47584c2128c142871604bfb06fb332385e5fc2812f15293577 WatchSource:0}: Error finding container 14f87508b96abe47584c2128c142871604bfb06fb332385e5fc2812f15293577: Status 404 returned error can't find the container with id 14f87508b96abe47584c2128c142871604bfb06fb332385e5fc2812f15293577 Jul 21 21:54:08.340178 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.340093 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" event={"ID":"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b","Type":"ContainerStarted","Data":"14f87508b96abe47584c2128c142871604bfb06fb332385e5fc2812f15293577"} Jul 21 21:54:08.604703 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.604619 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl"] Jul 21 21:54:08.609244 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.609220 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" Jul 21 21:54:08.612437 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.612414 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 21 21:54:08.613749 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.613731 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 21 21:54:08.613890 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.613831 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 21 21:54:08.614024 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.614010 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7jjg6\"" Jul 21 21:54:08.621271 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.620266 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl"] Jul 21 21:54:08.729303 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.729266 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6f69b996-0878-41ef-9a1c-d7d1726e64fb-manager-config\") pod \"lws-controller-manager-d89c66ff-94xzl\" (UID: \"6f69b996-0878-41ef-9a1c-d7d1726e64fb\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" Jul 21 21:54:08.729727 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.729328 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzjp7\" (UniqueName: \"kubernetes.io/projected/6f69b996-0878-41ef-9a1c-d7d1726e64fb-kube-api-access-nzjp7\") pod \"lws-controller-manager-d89c66ff-94xzl\" (UID: \"6f69b996-0878-41ef-9a1c-d7d1726e64fb\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" Jul 21 21:54:08.729727 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.729388 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f69b996-0878-41ef-9a1c-d7d1726e64fb-cert\") pod \"lws-controller-manager-d89c66ff-94xzl\" (UID: \"6f69b996-0878-41ef-9a1c-d7d1726e64fb\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" Jul 21 21:54:08.729727 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.729430 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f69b996-0878-41ef-9a1c-d7d1726e64fb-metrics-cert\") pod \"lws-controller-manager-d89c66ff-94xzl\" (UID: \"6f69b996-0878-41ef-9a1c-d7d1726e64fb\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" Jul 21 21:54:08.830575 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.830537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6f69b996-0878-41ef-9a1c-d7d1726e64fb-manager-config\") pod \"lws-controller-manager-d89c66ff-94xzl\" (UID: \"6f69b996-0878-41ef-9a1c-d7d1726e64fb\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" Jul 21 21:54:08.830737 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.830591 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjp7\" (UniqueName: \"kubernetes.io/projected/6f69b996-0878-41ef-9a1c-d7d1726e64fb-kube-api-access-nzjp7\") pod \"lws-controller-manager-d89c66ff-94xzl\" (UID: \"6f69b996-0878-41ef-9a1c-d7d1726e64fb\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" Jul 21 21:54:08.830737 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.830620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f69b996-0878-41ef-9a1c-d7d1726e64fb-cert\") pod \"lws-controller-manager-d89c66ff-94xzl\" (UID: \"6f69b996-0878-41ef-9a1c-d7d1726e64fb\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" Jul 21 21:54:08.830737 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.830645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f69b996-0878-41ef-9a1c-d7d1726e64fb-metrics-cert\") pod \"lws-controller-manager-d89c66ff-94xzl\" (UID: \"6f69b996-0878-41ef-9a1c-d7d1726e64fb\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" Jul 21 21:54:08.831155 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.831136 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6f69b996-0878-41ef-9a1c-d7d1726e64fb-manager-config\") pod \"lws-controller-manager-d89c66ff-94xzl\" (UID: \"6f69b996-0878-41ef-9a1c-d7d1726e64fb\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" Jul 21 21:54:08.833184 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.833161 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f69b996-0878-41ef-9a1c-d7d1726e64fb-metrics-cert\") pod \"lws-controller-manager-d89c66ff-94xzl\" (UID: \"6f69b996-0878-41ef-9a1c-d7d1726e64fb\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" Jul 21 21:54:08.833779 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.833760 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f69b996-0878-41ef-9a1c-d7d1726e64fb-cert\") pod \"lws-controller-manager-d89c66ff-94xzl\" (UID: \"6f69b996-0878-41ef-9a1c-d7d1726e64fb\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" Jul 21 21:54:08.843820 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.843801 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzjp7\" (UniqueName: \"kubernetes.io/projected/6f69b996-0878-41ef-9a1c-d7d1726e64fb-kube-api-access-nzjp7\") pod \"lws-controller-manager-d89c66ff-94xzl\" (UID: \"6f69b996-0878-41ef-9a1c-d7d1726e64fb\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" Jul 21 21:54:08.919146 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:08.919053 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" Jul 21 21:54:09.074709 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:09.074680 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl"] Jul 21 21:54:09.077250 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:54:09.077219 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f69b996_0878_41ef_9a1c_d7d1726e64fb.slice/crio-3f7b987f397440216ec07a559e91061fb421d96dda4918dd328a4fe36ff24378 WatchSource:0}: Error finding container 3f7b987f397440216ec07a559e91061fb421d96dda4918dd328a4fe36ff24378: Status 404 returned error can't find the container with id 3f7b987f397440216ec07a559e91061fb421d96dda4918dd328a4fe36ff24378 Jul 21 21:54:09.347233 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:09.347177 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" event={"ID":"6f69b996-0878-41ef-9a1c-d7d1726e64fb","Type":"ContainerStarted","Data":"3f7b987f397440216ec07a559e91061fb421d96dda4918dd328a4fe36ff24378"} Jul 21 21:54:11.427950 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:11.427911 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rml2h"] Jul 21 21:54:11.431647 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:11.431625 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rml2h" Jul 21 21:54:11.435137 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:11.435117 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-nxvx6\"" Jul 21 21:54:11.435737 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:11.435717 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 21 21:54:11.442099 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:11.442080 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rml2h"] Jul 21 21:54:11.497466 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:11.497354 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 21:54:11.497571 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:11.497516 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 21:54:11.497571 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:11.497567 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 21:54:11.557446 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:11.557422 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b27vp\" (UniqueName: \"kubernetes.io/projected/634a3cd8-16b4-417b-82ec-1f63d8c65dd7-kube-api-access-b27vp\") pod \"kserve-controller-manager-d6f4b65b8-rml2h\" (UID: \"634a3cd8-16b4-417b-82ec-1f63d8c65dd7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rml2h" Jul 21 21:54:11.557557 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:11.557457 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/634a3cd8-16b4-417b-82ec-1f63d8c65dd7-cert\") pod \"kserve-controller-manager-d6f4b65b8-rml2h\" (UID: \"634a3cd8-16b4-417b-82ec-1f63d8c65dd7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rml2h" Jul 21 21:54:11.658959 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:11.658924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b27vp\" (UniqueName: \"kubernetes.io/projected/634a3cd8-16b4-417b-82ec-1f63d8c65dd7-kube-api-access-b27vp\") pod \"kserve-controller-manager-d6f4b65b8-rml2h\" (UID: \"634a3cd8-16b4-417b-82ec-1f63d8c65dd7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rml2h" Jul 21 21:54:11.659146 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:11.658970 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/634a3cd8-16b4-417b-82ec-1f63d8c65dd7-cert\") pod \"kserve-controller-manager-d6f4b65b8-rml2h\" (UID: \"634a3cd8-16b4-417b-82ec-1f63d8c65dd7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rml2h" Jul 21 21:54:11.659216 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:54:11.659144 2580 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 21 21:54:11.659267 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:54:11.659215 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/634a3cd8-16b4-417b-82ec-1f63d8c65dd7-cert podName:634a3cd8-16b4-417b-82ec-1f63d8c65dd7 nodeName:}" failed. No retries permitted until 2026-07-21 21:54:12.159194101 +0000 UTC m=+329.922428343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/634a3cd8-16b4-417b-82ec-1f63d8c65dd7-cert") pod "kserve-controller-manager-d6f4b65b8-rml2h" (UID: "634a3cd8-16b4-417b-82ec-1f63d8c65dd7") : secret "kserve-webhook-server-cert" not found Jul 21 21:54:11.673621 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:11.673578 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b27vp\" (UniqueName: \"kubernetes.io/projected/634a3cd8-16b4-417b-82ec-1f63d8c65dd7-kube-api-access-b27vp\") pod \"kserve-controller-manager-d6f4b65b8-rml2h\" (UID: \"634a3cd8-16b4-417b-82ec-1f63d8c65dd7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rml2h" Jul 21 21:54:12.164575 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:12.164534 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/634a3cd8-16b4-417b-82ec-1f63d8c65dd7-cert\") pod \"kserve-controller-manager-d6f4b65b8-rml2h\" (UID: \"634a3cd8-16b4-417b-82ec-1f63d8c65dd7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rml2h" Jul 21 21:54:12.167088 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:12.167064 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/634a3cd8-16b4-417b-82ec-1f63d8c65dd7-cert\") pod \"kserve-controller-manager-d6f4b65b8-rml2h\" (UID: \"634a3cd8-16b4-417b-82ec-1f63d8c65dd7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rml2h" Jul 21 21:54:12.344748 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:12.344710 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rml2h" Jul 21 21:54:12.366926 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:12.366860 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" event={"ID":"d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b","Type":"ContainerStarted","Data":"fd5a32d652fb8e1c2da69d6cb498e3b8d0a850739712f623b9b733f75c16710f"} Jul 21 21:54:12.368519 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:12.368489 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" event={"ID":"6f69b996-0878-41ef-9a1c-d7d1726e64fb","Type":"ContainerStarted","Data":"c99c175554a4c5e7d6ed2cf647cc0347563debda19d21fdece331552fcb981b1"} Jul 21 21:54:12.368650 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:12.368572 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" Jul 21 21:54:12.391165 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:12.391031 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" podStartSLOduration=2.0819046 podStartE2EDuration="5.391008422s" totalImagesPullingTime="3.309103822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:54:12.390406708 +0000 UTC m=+330.153640967" watchObservedRunningTime="2026-07-21 21:54:12.391008422 +0000 UTC m=+330.154242684" Jul 21 21:54:12.411761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:12.409444 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" podStartSLOduration=1.992783811 podStartE2EDuration="4.409421293s" totalImagesPullingTime="2.416637482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:54:12.408669838 +0000 UTC m=+330.171904100" watchObservedRunningTime="2026-07-21 21:54:12.409421293 +0000 UTC m=+330.172655556" Jul 21 21:54:12.501315 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:12.501290 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rml2h"] Jul 21 21:54:12.503127 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:54:12.503102 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634a3cd8_16b4_417b_82ec_1f63d8c65dd7.slice/crio-2da6737c6af5c5d4d06a4828903215136d60f85c63112e6abbf152a51e457719 WatchSource:0}: Error finding container 2da6737c6af5c5d4d06a4828903215136d60f85c63112e6abbf152a51e457719: Status 404 returned error can't find the container with id 2da6737c6af5c5d4d06a4828903215136d60f85c63112e6abbf152a51e457719 Jul 21 21:54:13.026991 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:13.026954 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:13.031817 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:13.031791 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:13.375075 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:13.374982 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rml2h" event={"ID":"634a3cd8-16b4-417b-82ec-1f63d8c65dd7","Type":"ContainerStarted","Data":"2da6737c6af5c5d4d06a4828903215136d60f85c63112e6abbf152a51e457719"} Jul 21 21:54:13.375287 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:13.375263 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:13.376350 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:13.376327 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vpjh7" Jul 21 21:54:16.390586 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:16.390545 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rml2h" event={"ID":"634a3cd8-16b4-417b-82ec-1f63d8c65dd7","Type":"ContainerStarted","Data":"6b802b4efb1efe7f5c305c7c139142ab5824e03860229add658bc93f49abe483"} Jul 21 21:54:16.391015 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:16.390765 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rml2h" Jul 21 21:54:16.409113 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:16.409071 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rml2h" podStartSLOduration=2.603372616 podStartE2EDuration="5.409056447s" totalImagesPullingTime="2.805683831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:54:16.407469161 +0000 UTC m=+334.170703423" watchObservedRunningTime="2026-07-21 21:54:16.409056447 +0000 UTC m=+334.172290711" Jul 21 21:54:23.377367 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:23.377337 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-d89c66ff-94xzl" Jul 21 21:54:47.399981 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:47.399902 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rml2h" Jul 21 21:54:59.599642 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:59.599606 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7gv7v"] Jul 21 21:54:59.606690 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:59.606662 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7gv7v" Jul 21 21:54:59.609415 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:59.609391 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 21:54:59.609554 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:59.609494 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-84lqh\"" Jul 21 21:54:59.610269 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:59.610253 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 21:54:59.621652 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:59.621628 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7gv7v"] Jul 21 21:54:59.671286 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:59.671254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr8vf\" (UniqueName: \"kubernetes.io/projected/ee76607b-d977-4572-9cf8-1eeda281e068-kube-api-access-mr8vf\") pod \"kuadrant-operator-catalog-7gv7v\" (UID: \"ee76607b-d977-4572-9cf8-1eeda281e068\") " pod="kuadrant-system/kuadrant-operator-catalog-7gv7v" Jul 21 21:54:59.772548 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:59.772509 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr8vf\" (UniqueName: \"kubernetes.io/projected/ee76607b-d977-4572-9cf8-1eeda281e068-kube-api-access-mr8vf\") pod \"kuadrant-operator-catalog-7gv7v\" (UID: \"ee76607b-d977-4572-9cf8-1eeda281e068\") " pod="kuadrant-system/kuadrant-operator-catalog-7gv7v" Jul 21 21:54:59.781964 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:59.781933 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr8vf\" (UniqueName: \"kubernetes.io/projected/ee76607b-d977-4572-9cf8-1eeda281e068-kube-api-access-mr8vf\") pod \"kuadrant-operator-catalog-7gv7v\" (UID: \"ee76607b-d977-4572-9cf8-1eeda281e068\") " pod="kuadrant-system/kuadrant-operator-catalog-7gv7v" Jul 21 21:54:59.917650 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:59.917567 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7gv7v" Jul 21 21:54:59.939667 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:54:59.939636 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7gv7v"] Jul 21 21:55:00.057870 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:00.057806 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7gv7v"] Jul 21 21:55:00.060113 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:55:00.060089 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee76607b_d977_4572_9cf8_1eeda281e068.slice/crio-3707cd05fa74f163b31af253861fb491faaf00f1fababe9b1fa006677c93cdad WatchSource:0}: Error finding container 3707cd05fa74f163b31af253861fb491faaf00f1fababe9b1fa006677c93cdad: Status 404 returned error can't find the container with id 3707cd05fa74f163b31af253861fb491faaf00f1fababe9b1fa006677c93cdad Jul 21 21:55:00.147566 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:00.147528 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j69kr"] Jul 21 21:55:00.150641 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:00.150618 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-j69kr" Jul 21 21:55:00.160102 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:00.160077 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j69kr"] Jul 21 21:55:00.277809 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:00.277779 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dggzr\" (UniqueName: \"kubernetes.io/projected/fb55a8dc-fe35-44ae-9a0e-4b9178401a85-kube-api-access-dggzr\") pod \"kuadrant-operator-catalog-j69kr\" (UID: \"fb55a8dc-fe35-44ae-9a0e-4b9178401a85\") " pod="kuadrant-system/kuadrant-operator-catalog-j69kr" Jul 21 21:55:00.378522 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:00.378483 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dggzr\" (UniqueName: \"kubernetes.io/projected/fb55a8dc-fe35-44ae-9a0e-4b9178401a85-kube-api-access-dggzr\") pod \"kuadrant-operator-catalog-j69kr\" (UID: \"fb55a8dc-fe35-44ae-9a0e-4b9178401a85\") " pod="kuadrant-system/kuadrant-operator-catalog-j69kr" Jul 21 21:55:00.388060 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:00.388042 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dggzr\" (UniqueName: \"kubernetes.io/projected/fb55a8dc-fe35-44ae-9a0e-4b9178401a85-kube-api-access-dggzr\") pod \"kuadrant-operator-catalog-j69kr\" (UID: \"fb55a8dc-fe35-44ae-9a0e-4b9178401a85\") " pod="kuadrant-system/kuadrant-operator-catalog-j69kr" Jul 21 21:55:00.461176 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:00.461147 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-j69kr" Jul 21 21:55:00.589550 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:00.589516 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7gv7v" event={"ID":"ee76607b-d977-4572-9cf8-1eeda281e068","Type":"ContainerStarted","Data":"3707cd05fa74f163b31af253861fb491faaf00f1fababe9b1fa006677c93cdad"} Jul 21 21:55:00.606653 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:00.606565 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j69kr"] Jul 21 21:55:00.608747 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:55:00.608720 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb55a8dc_fe35_44ae_9a0e_4b9178401a85.slice/crio-62539159f12111e6b8da0bfcef58783c905d28a087dc0e1e0f8b5634ff04f525 WatchSource:0}: Error finding container 62539159f12111e6b8da0bfcef58783c905d28a087dc0e1e0f8b5634ff04f525: Status 404 returned error can't find the container with id 62539159f12111e6b8da0bfcef58783c905d28a087dc0e1e0f8b5634ff04f525 Jul 21 21:55:01.595837 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:01.595804 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-j69kr" event={"ID":"fb55a8dc-fe35-44ae-9a0e-4b9178401a85","Type":"ContainerStarted","Data":"62539159f12111e6b8da0bfcef58783c905d28a087dc0e1e0f8b5634ff04f525"} Jul 21 21:55:02.601945 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:02.601897 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-j69kr" event={"ID":"fb55a8dc-fe35-44ae-9a0e-4b9178401a85","Type":"ContainerStarted","Data":"6a417296eadfe40846724f2d6d1d233adec438e4bb05448e591da9f62e1c08b4"} Jul 21 21:55:02.603303 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:02.603278 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7gv7v" event={"ID":"ee76607b-d977-4572-9cf8-1eeda281e068","Type":"ContainerStarted","Data":"3a0121b6d3f6ea58710b312927f4e5ede7e6a3306f2b722441825968fda8c864"} Jul 21 21:55:02.603394 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:02.603302 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-7gv7v" podUID="ee76607b-d977-4572-9cf8-1eeda281e068" containerName="registry-server" containerID="cri-o://3a0121b6d3f6ea58710b312927f4e5ede7e6a3306f2b722441825968fda8c864" gracePeriod=2 Jul 21 21:55:02.620433 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:02.620390 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-j69kr" podStartSLOduration=0.774714508 podStartE2EDuration="2.620355846s" totalImagesPullingTime="1.845641338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:55:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:55:02.618469947 +0000 UTC m=+380.381704209" watchObservedRunningTime="2026-07-21 21:55:02.620355846 +0000 UTC m=+380.383590106" Jul 21 21:55:02.634666 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:02.634623 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-7gv7v" podStartSLOduration=1.2411581919999999 podStartE2EDuration="3.634607519s" totalImagesPullingTime="2.393449327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:54:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:55:02.634124649 +0000 UTC m=+380.397358923" watchObservedRunningTime="2026-07-21 21:55:02.634607519 +0000 UTC m=+380.397841780" Jul 21 21:55:02.851322 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:02.851300 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7gv7v_ee76607b-d977-4572-9cf8-1eeda281e068/registry-server/0.log" Jul 21 21:55:02.853714 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:02.853699 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7gv7v" Jul 21 21:55:03.003502 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:03.003467 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mr8vf\" (UniqueName: \"kubernetes.io/projected/ee76607b-d977-4572-9cf8-1eeda281e068-kube-api-access-mr8vf\") pod \"ee76607b-d977-4572-9cf8-1eeda281e068\" (UID: \"ee76607b-d977-4572-9cf8-1eeda281e068\") " Jul 21 21:55:03.005785 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:03.005761 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee76607b-d977-4572-9cf8-1eeda281e068-kube-api-access-mr8vf" (OuterVolumeSpecName: "kube-api-access-mr8vf") pod "ee76607b-d977-4572-9cf8-1eeda281e068" (UID: "ee76607b-d977-4572-9cf8-1eeda281e068"). InnerVolumeSpecName "kube-api-access-mr8vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:55:03.104961 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:03.104923 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr8vf\" (UniqueName: \"kubernetes.io/projected/ee76607b-d977-4572-9cf8-1eeda281e068-kube-api-access-mr8vf\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:55:03.611571 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:03.611540 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7gv7v_ee76607b-d977-4572-9cf8-1eeda281e068/registry-server/0.log" Jul 21 21:55:03.611994 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:03.611582 2580 generic.go:356] "Generic (PLEG): container finished" podID="ee76607b-d977-4572-9cf8-1eeda281e068" containerID="3a0121b6d3f6ea58710b312927f4e5ede7e6a3306f2b722441825968fda8c864" exitCode=1 Jul 21 21:55:03.611994 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:03.611616 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7gv7v" event={"ID":"ee76607b-d977-4572-9cf8-1eeda281e068","Type":"ContainerDied","Data":"3a0121b6d3f6ea58710b312927f4e5ede7e6a3306f2b722441825968fda8c864"} Jul 21 21:55:03.611994 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:03.611659 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7gv7v" Jul 21 21:55:03.611994 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:03.611712 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7gv7v" event={"ID":"ee76607b-d977-4572-9cf8-1eeda281e068","Type":"ContainerDied","Data":"3707cd05fa74f163b31af253861fb491faaf00f1fababe9b1fa006677c93cdad"} Jul 21 21:55:03.611994 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:03.611742 2580 scope.go:117] "RemoveContainer" containerID="3a0121b6d3f6ea58710b312927f4e5ede7e6a3306f2b722441825968fda8c864" Jul 21 21:55:03.621567 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:03.621549 2580 scope.go:117] "RemoveContainer" containerID="3a0121b6d3f6ea58710b312927f4e5ede7e6a3306f2b722441825968fda8c864" Jul 21 21:55:03.621872 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:03.621853 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0121b6d3f6ea58710b312927f4e5ede7e6a3306f2b722441825968fda8c864\": container with ID starting with 3a0121b6d3f6ea58710b312927f4e5ede7e6a3306f2b722441825968fda8c864 not found: ID does not exist" containerID="3a0121b6d3f6ea58710b312927f4e5ede7e6a3306f2b722441825968fda8c864" Jul 21 21:55:03.621929 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:03.621882 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0121b6d3f6ea58710b312927f4e5ede7e6a3306f2b722441825968fda8c864"} err="failed to get container status \"3a0121b6d3f6ea58710b312927f4e5ede7e6a3306f2b722441825968fda8c864\": rpc error: code = NotFound desc = could not find container \"3a0121b6d3f6ea58710b312927f4e5ede7e6a3306f2b722441825968fda8c864\": container with ID starting with 3a0121b6d3f6ea58710b312927f4e5ede7e6a3306f2b722441825968fda8c864 not found: ID does not exist" Jul 21 21:55:03.634456 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:03.634407 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7gv7v"] Jul 21 21:55:03.636166 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:03.636144 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7gv7v"] Jul 21 21:55:04.779582 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:04.779552 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee76607b-d977-4572-9cf8-1eeda281e068" path="/var/lib/kubelet/pods/ee76607b-d977-4572-9cf8-1eeda281e068/volumes" Jul 21 21:55:10.461934 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:10.461894 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-j69kr" Jul 21 21:55:10.462426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:10.462002 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-j69kr" Jul 21 21:55:10.485202 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:10.485165 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-j69kr" Jul 21 21:55:10.668702 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:10.668673 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-j69kr" Jul 21 21:55:21.307602 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.307571 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7499db7949-q2wng"] Jul 21 21:55:21.308050 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.307975 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee76607b-d977-4572-9cf8-1eeda281e068" containerName="registry-server" Jul 21 21:55:21.308050 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.307987 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee76607b-d977-4572-9cf8-1eeda281e068" containerName="registry-server" Jul 21 21:55:21.308050 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.308047 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee76607b-d977-4572-9cf8-1eeda281e068" containerName="registry-server" Jul 21 21:55:21.314893 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.313361 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.317420 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.317296 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 21 21:55:21.317420 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.317308 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2fvdp\"" Jul 21 21:55:21.317420 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.317322 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 21 21:55:21.317420 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.317341 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 21 21:55:21.317420 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.317295 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 21 21:55:21.317776 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.317575 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 21:55:21.317776 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.317610 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 21:55:21.317776 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.317611 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 21 21:55:21.327402 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.327355 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 21 21:55:21.328148 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.328126 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7499db7949-q2wng"] Jul 21 21:55:21.469917 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.469882 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-trusted-ca-bundle\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.470090 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.469934 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-console-oauth-config\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.470090 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.469957 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-service-ca\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.470090 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.469977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-oauth-serving-cert\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.470090 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.469994 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px8fv\" (UniqueName: \"kubernetes.io/projected/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-kube-api-access-px8fv\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.470222 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.470114 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-console-config\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.470222 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.470154 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-console-serving-cert\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.570695 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.570616 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-console-config\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.570695 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.570658 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-console-serving-cert\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.570695 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.570683 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-trusted-ca-bundle\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.570949 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.570717 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-console-oauth-config\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.570949 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.570737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-service-ca\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.570949 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.570755 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-oauth-serving-cert\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.570949 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.570771 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px8fv\" (UniqueName: \"kubernetes.io/projected/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-kube-api-access-px8fv\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.571516 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.571496 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-service-ca\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.571683 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.571667 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-trusted-ca-bundle\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.571764 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.571670 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-console-config\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.571764 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.571719 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-oauth-serving-cert\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.573265 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.573239 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-console-oauth-config\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.573495 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.573475 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-console-serving-cert\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.582426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.582401 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px8fv\" (UniqueName: \"kubernetes.io/projected/d9882f85-b8e1-45b9-8083-9cf6f9cf756c-kube-api-access-px8fv\") pod \"console-7499db7949-q2wng\" (UID: \"d9882f85-b8e1-45b9-8083-9cf6f9cf756c\") " pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.624700 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.624667 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:21.760810 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:21.760785 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7499db7949-q2wng"] Jul 21 21:55:21.763179 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:55:21.763153 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9882f85_b8e1_45b9_8083_9cf6f9cf756c.slice/crio-384062193b3f4e10d256005ea8c3ca55a21ad6a5eba01a8f4757617d247755f7 WatchSource:0}: Error finding container 384062193b3f4e10d256005ea8c3ca55a21ad6a5eba01a8f4757617d247755f7: Status 404 returned error can't find the container with id 384062193b3f4e10d256005ea8c3ca55a21ad6a5eba01a8f4757617d247755f7 Jul 21 21:55:22.702096 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:22.702054 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7499db7949-q2wng" event={"ID":"d9882f85-b8e1-45b9-8083-9cf6f9cf756c","Type":"ContainerStarted","Data":"65ce45cdcfe5dc001055ead7b07e6d5985a36e9103db2dc0855074174e2b461a"} Jul 21 21:55:22.702502 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:22.702110 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7499db7949-q2wng" event={"ID":"d9882f85-b8e1-45b9-8083-9cf6f9cf756c","Type":"ContainerStarted","Data":"384062193b3f4e10d256005ea8c3ca55a21ad6a5eba01a8f4757617d247755f7"} Jul 21 21:55:22.723878 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:22.723829 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7499db7949-q2wng" podStartSLOduration=1.72381422 podStartE2EDuration="1.72381422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:55:22.721603214 +0000 UTC m=+400.484837475" watchObservedRunningTime="2026-07-21 21:55:22.72381422 +0000 UTC m=+400.487048480" Jul 21 21:55:25.062588 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:25.062555 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq"] Jul 21 21:55:25.067454 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:25.067430 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" Jul 21 21:55:25.070278 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:25.070256 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-44pmh\"" Jul 21 21:55:25.076053 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:25.076030 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq"] Jul 21 21:55:25.097209 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:25.097189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxnzw\" (UniqueName: \"kubernetes.io/projected/29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a-kube-api-access-bxnzw\") pod \"limitador-operator-controller-manager-859d7b78bd-rl2xq\" (UID: \"29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" Jul 21 21:55:25.197677 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:25.197648 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxnzw\" (UniqueName: \"kubernetes.io/projected/29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a-kube-api-access-bxnzw\") pod \"limitador-operator-controller-manager-859d7b78bd-rl2xq\" (UID: \"29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" Jul 21 21:55:25.212001 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:25.211980 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxnzw\" (UniqueName: \"kubernetes.io/projected/29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a-kube-api-access-bxnzw\") pod \"limitador-operator-controller-manager-859d7b78bd-rl2xq\" (UID: \"29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" Jul 21 21:55:25.378598 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:25.378510 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" Jul 21 21:55:25.536761 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:25.536737 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq"] Jul 21 21:55:25.538748 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:55:25.538723 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29095fb9_0f81_45b2_a5c2_1cc0cfc7b71a.slice/crio-bbad921f6745db9305143a27d43b030ab18b9ab2084e64cadc380414e29a762a WatchSource:0}: Error finding container bbad921f6745db9305143a27d43b030ab18b9ab2084e64cadc380414e29a762a: Status 404 returned error can't find the container with id bbad921f6745db9305143a27d43b030ab18b9ab2084e64cadc380414e29a762a Jul 21 21:55:25.718538 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:25.718451 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" event={"ID":"29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a","Type":"ContainerStarted","Data":"bbad921f6745db9305143a27d43b030ab18b9ab2084e64cadc380414e29a762a"} Jul 21 21:55:27.729602 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:27.729567 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" event={"ID":"29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a","Type":"ContainerStarted","Data":"9fc32be1a1246315b670e5440932254baff5ef8042de033958be38868b680fbe"} Jul 21 21:55:27.729995 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:27.729703 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" Jul 21 21:55:27.753931 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:27.753885 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" podStartSLOduration=1.069043213 podStartE2EDuration="2.753868925s" totalImagesPullingTime="1.684825712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:55:27.752187606 +0000 UTC m=+405.515421867" watchObservedRunningTime="2026-07-21 21:55:27.753868925 +0000 UTC m=+405.517103186" Jul 21 21:55:31.625147 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:31.625114 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:31.625510 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:31.625209 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:31.629745 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:31.629722 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:31.751498 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:31.751474 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7499db7949-q2wng" Jul 21 21:55:34.519567 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:34.519536 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6"] Jul 21 21:55:34.522834 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:34.522818 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:34.525498 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:34.525474 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tsvj5\"" Jul 21 21:55:34.535837 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:34.535814 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6"] Jul 21 21:55:34.580285 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:34.580252 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r44w\" (UniqueName: \"kubernetes.io/projected/a965e5de-653e-4302-8176-45e0229ed359-kube-api-access-7r44w\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wz2z6\" (UID: \"a965e5de-653e-4302-8176-45e0229ed359\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:34.580416 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:34.580308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a965e5de-653e-4302-8176-45e0229ed359-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wz2z6\" (UID: \"a965e5de-653e-4302-8176-45e0229ed359\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:34.681330 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:34.681281 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r44w\" (UniqueName: \"kubernetes.io/projected/a965e5de-653e-4302-8176-45e0229ed359-kube-api-access-7r44w\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wz2z6\" (UID: \"a965e5de-653e-4302-8176-45e0229ed359\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:34.681475 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:34.681353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a965e5de-653e-4302-8176-45e0229ed359-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wz2z6\" (UID: \"a965e5de-653e-4302-8176-45e0229ed359\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:34.681696 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:34.681680 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a965e5de-653e-4302-8176-45e0229ed359-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wz2z6\" (UID: \"a965e5de-653e-4302-8176-45e0229ed359\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:34.689940 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:34.689914 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r44w\" (UniqueName: \"kubernetes.io/projected/a965e5de-653e-4302-8176-45e0229ed359-kube-api-access-7r44w\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wz2z6\" (UID: \"a965e5de-653e-4302-8176-45e0229ed359\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:34.832765 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:34.832693 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:34.974182 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:34.974144 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6"] Jul 21 21:55:34.978558 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:55:34.978516 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda965e5de_653e_4302_8176_45e0229ed359.slice/crio-fc97ac5d12d487ba50d461073cf4f1d8c3de32fd017dccf36f4952782f49c087 WatchSource:0}: Error finding container fc97ac5d12d487ba50d461073cf4f1d8c3de32fd017dccf36f4952782f49c087: Status 404 returned error can't find the container with id fc97ac5d12d487ba50d461073cf4f1d8c3de32fd017dccf36f4952782f49c087 Jul 21 21:55:35.769869 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:35.769793 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" event={"ID":"a965e5de-653e-4302-8176-45e0229ed359","Type":"ContainerStarted","Data":"fc97ac5d12d487ba50d461073cf4f1d8c3de32fd017dccf36f4952782f49c087"} Jul 21 21:55:38.736808 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:38.736783 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" Jul 21 21:55:40.797357 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:40.797321 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" event={"ID":"a965e5de-653e-4302-8176-45e0229ed359","Type":"ContainerStarted","Data":"2342274d7ab865d53b6a0ce9fc3a91cf79063c49d33ad6fa5df1a69472cc4e05"} Jul 21 21:55:40.797733 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:40.797522 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:40.819023 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:40.818983 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" podStartSLOduration=1.642436883 podStartE2EDuration="6.818969526s" totalImagesPullingTime="5.176532643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:55:40.816473564 +0000 UTC m=+418.579707847" watchObservedRunningTime="2026-07-21 21:55:40.818969526 +0000 UTC m=+418.582203787" Jul 21 21:55:51.805974 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:51.805944 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:52.745271 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:52.745229 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j"] Jul 21 21:55:52.748770 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:52.748746 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" Jul 21 21:55:52.766125 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:52.766101 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j"] Jul 21 21:55:52.855474 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:52.855429 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv5qv\" (UniqueName: \"kubernetes.io/projected/9b7dcd4e-a265-487c-a47a-7fdc9753c119-kube-api-access-zv5qv\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rch5j\" (UID: \"9b7dcd4e-a265-487c-a47a-7fdc9753c119\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" Jul 21 21:55:52.856055 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:52.856032 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b7dcd4e-a265-487c-a47a-7fdc9753c119-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rch5j\" (UID: \"9b7dcd4e-a265-487c-a47a-7fdc9753c119\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" Jul 21 21:55:52.956968 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:52.956917 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv5qv\" (UniqueName: \"kubernetes.io/projected/9b7dcd4e-a265-487c-a47a-7fdc9753c119-kube-api-access-zv5qv\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rch5j\" (UID: \"9b7dcd4e-a265-487c-a47a-7fdc9753c119\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" Jul 21 21:55:52.957158 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:52.957004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b7dcd4e-a265-487c-a47a-7fdc9753c119-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rch5j\" (UID: \"9b7dcd4e-a265-487c-a47a-7fdc9753c119\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" Jul 21 21:55:52.957350 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:52.957333 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b7dcd4e-a265-487c-a47a-7fdc9753c119-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rch5j\" (UID: \"9b7dcd4e-a265-487c-a47a-7fdc9753c119\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" Jul 21 21:55:52.968716 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:52.968683 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv5qv\" (UniqueName: \"kubernetes.io/projected/9b7dcd4e-a265-487c-a47a-7fdc9753c119-kube-api-access-zv5qv\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rch5j\" (UID: \"9b7dcd4e-a265-487c-a47a-7fdc9753c119\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" Jul 21 21:55:53.059622 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.059526 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" Jul 21 21:55:53.199284 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.199260 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j"] Jul 21 21:55:53.201674 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:55:53.201649 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7dcd4e_a265_487c_a47a_7fdc9753c119.slice/crio-ba51f068c55170e07dcf8c3975fddd7b9397e6246e5c9ffdb6c6c3373cf32c7d WatchSource:0}: Error finding container ba51f068c55170e07dcf8c3975fddd7b9397e6246e5c9ffdb6c6c3373cf32c7d: Status 404 returned error can't find the container with id ba51f068c55170e07dcf8c3975fddd7b9397e6246e5c9ffdb6c6c3373cf32c7d Jul 21 21:55:53.496675 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.496644 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j"] Jul 21 21:55:53.507304 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.507277 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j"] Jul 21 21:55:53.516424 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.516397 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6"] Jul 21 21:55:53.516637 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.516589 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" podUID="a965e5de-653e-4302-8176-45e0229ed359" containerName="manager" containerID="cri-o://2342274d7ab865d53b6a0ce9fc3a91cf79063c49d33ad6fa5df1a69472cc4e05" gracePeriod=2 Jul 21 21:55:53.520760 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.520731 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr"] Jul 21 21:55:53.521201 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.521183 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b7dcd4e-a265-487c-a47a-7fdc9753c119" containerName="manager" Jul 21 21:55:53.521201 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.521203 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b7dcd4e-a265-487c-a47a-7fdc9753c119" containerName="manager" Jul 21 21:55:53.521321 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.521265 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b7dcd4e-a265-487c-a47a-7fdc9753c119" containerName="manager" Jul 21 21:55:53.523444 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.523427 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr" Jul 21 21:55:53.528136 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.528116 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6"] Jul 21 21:55:53.530634 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:53.530582 2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"a965e5de-653e-4302-8176-45e0229ed359\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-wz2z6\": pods \"kuadrant-operator-controller-manager-8684c9ff49-wz2z6\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:53.537760 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.537728 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr"] Jul 21 21:55:53.541992 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.541969 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v"] Jul 21 21:55:53.542530 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.542514 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a965e5de-653e-4302-8176-45e0229ed359" containerName="manager" Jul 21 21:55:53.542634 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.542533 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a965e5de-653e-4302-8176-45e0229ed359" containerName="manager" Jul 21 21:55:53.542634 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.542621 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a965e5de-653e-4302-8176-45e0229ed359" containerName="manager" Jul 21 21:55:53.544401 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.544384 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq"] Jul 21 21:55:53.544496 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.544483 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v" Jul 21 21:55:53.544599 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.544579 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" podUID="29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a" containerName="manager" containerID="cri-o://9fc32be1a1246315b670e5440932254baff5ef8042de033958be38868b680fbe" gracePeriod=2 Jul 21 21:55:53.552187 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:53.552154 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wz2z6\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="a965e5de-653e-4302-8176-45e0229ed359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:53.554446 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:53.554420 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rl2xq\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" Jul 21 21:55:53.556157 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.556045 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq"] Jul 21 21:55:53.558438 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.558356 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v"] Jul 21 21:55:53.576476 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:53.576437 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wz2z6\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="a965e5de-653e-4302-8176-45e0229ed359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:53.663141 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.663110 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6jmj\" (UniqueName: \"kubernetes.io/projected/16ab49c4-8ca9-4aee-853c-73c55cb1b843-kube-api-access-n6jmj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9n29v\" (UID: \"16ab49c4-8ca9-4aee-853c-73c55cb1b843\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v" Jul 21 21:55:53.663292 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.663248 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16ab49c4-8ca9-4aee-853c-73c55cb1b843-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9n29v\" (UID: \"16ab49c4-8ca9-4aee-853c-73c55cb1b843\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v" Jul 21 21:55:53.663355 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.663300 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4vfz\" (UniqueName: \"kubernetes.io/projected/5ea578df-97e2-4f8e-acd7-8117b90f2175-kube-api-access-t4vfz\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hmqmr\" (UID: \"5ea578df-97e2-4f8e-acd7-8117b90f2175\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr" Jul 21 21:55:53.663484 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.663452 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ea578df-97e2-4f8e-acd7-8117b90f2175-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hmqmr\" (UID: \"5ea578df-97e2-4f8e-acd7-8117b90f2175\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr" Jul 21 21:55:53.764233 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.764133 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ea578df-97e2-4f8e-acd7-8117b90f2175-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hmqmr\" (UID: \"5ea578df-97e2-4f8e-acd7-8117b90f2175\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr" Jul 21 21:55:53.764233 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.764183 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6jmj\" (UniqueName: \"kubernetes.io/projected/16ab49c4-8ca9-4aee-853c-73c55cb1b843-kube-api-access-n6jmj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9n29v\" (UID: \"16ab49c4-8ca9-4aee-853c-73c55cb1b843\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v" Jul 21 21:55:53.764492 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.764250 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16ab49c4-8ca9-4aee-853c-73c55cb1b843-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9n29v\" (UID: \"16ab49c4-8ca9-4aee-853c-73c55cb1b843\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v" Jul 21 21:55:53.764492 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.764288 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vfz\" (UniqueName: \"kubernetes.io/projected/5ea578df-97e2-4f8e-acd7-8117b90f2175-kube-api-access-t4vfz\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hmqmr\" (UID: \"5ea578df-97e2-4f8e-acd7-8117b90f2175\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr" Jul 21 21:55:53.764602 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.764589 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ea578df-97e2-4f8e-acd7-8117b90f2175-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hmqmr\" (UID: \"5ea578df-97e2-4f8e-acd7-8117b90f2175\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr" Jul 21 21:55:53.764758 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.764732 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16ab49c4-8ca9-4aee-853c-73c55cb1b843-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9n29v\" (UID: \"16ab49c4-8ca9-4aee-853c-73c55cb1b843\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v" Jul 21 21:55:53.773251 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.773225 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4vfz\" (UniqueName: \"kubernetes.io/projected/5ea578df-97e2-4f8e-acd7-8117b90f2175-kube-api-access-t4vfz\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hmqmr\" (UID: \"5ea578df-97e2-4f8e-acd7-8117b90f2175\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr" Jul 21 21:55:53.773881 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.773860 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6jmj\" (UniqueName: \"kubernetes.io/projected/16ab49c4-8ca9-4aee-853c-73c55cb1b843-kube-api-access-n6jmj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9n29v\" (UID: \"16ab49c4-8ca9-4aee-853c-73c55cb1b843\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v" Jul 21 21:55:53.778168 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.778149 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:53.780506 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:53.780478 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wz2z6\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="a965e5de-653e-4302-8176-45e0229ed359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:53.782734 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:53.782704 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rl2xq\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" Jul 21 21:55:53.786685 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.786667 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" Jul 21 21:55:53.788985 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:53.788962 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rl2xq\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" Jul 21 21:55:53.791051 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:53.791028 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wz2z6\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="a965e5de-653e-4302-8176-45e0229ed359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:53.861320 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:53.861290 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" container="manager" Jul 21 21:55:53.862319 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.862295 2580 generic.go:356] "Generic (PLEG): container finished" podID="a965e5de-653e-4302-8176-45e0229ed359" containerID="2342274d7ab865d53b6a0ce9fc3a91cf79063c49d33ad6fa5df1a69472cc4e05" exitCode=0 Jul 21 21:55:53.862494 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.862328 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:53.862494 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.862403 2580 scope.go:117] "RemoveContainer" containerID="2342274d7ab865d53b6a0ce9fc3a91cf79063c49d33ad6fa5df1a69472cc4e05" Jul 21 21:55:53.863358 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:53.863339 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rl2xq\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" Jul 21 21:55:53.863671 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.863651 2580 generic.go:356] "Generic (PLEG): container finished" podID="29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a" containerID="9fc32be1a1246315b670e5440932254baff5ef8042de033958be38868b680fbe" exitCode=0 Jul 21 21:55:53.863745 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.863706 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" Jul 21 21:55:53.864878 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.864851 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a965e5de-653e-4302-8176-45e0229ed359-extensions-socket-volume\") pod \"a965e5de-653e-4302-8176-45e0229ed359\" (UID: \"a965e5de-653e-4302-8176-45e0229ed359\") " Jul 21 21:55:53.864992 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.864922 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxnzw\" (UniqueName: \"kubernetes.io/projected/29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a-kube-api-access-bxnzw\") pod \"29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a\" (UID: \"29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a\") " Jul 21 21:55:53.865067 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.865046 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r44w\" (UniqueName: \"kubernetes.io/projected/a965e5de-653e-4302-8176-45e0229ed359-kube-api-access-7r44w\") pod \"a965e5de-653e-4302-8176-45e0229ed359\" (UID: \"a965e5de-653e-4302-8176-45e0229ed359\") " Jul 21 21:55:53.865212 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.865174 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a965e5de-653e-4302-8176-45e0229ed359-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a965e5de-653e-4302-8176-45e0229ed359" (UID: "a965e5de-653e-4302-8176-45e0229ed359"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:55:53.865437 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.865407 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a965e5de-653e-4302-8176-45e0229ed359-extensions-socket-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:55:53.866100 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:53.866065 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rch5j\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="9b7dcd4e-a265-487c-a47a-7fdc9753c119" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" Jul 21 21:55:53.867206 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.867184 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a-kube-api-access-bxnzw" (OuterVolumeSpecName: "kube-api-access-bxnzw") pod "29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a" (UID: "29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a"). InnerVolumeSpecName "kube-api-access-bxnzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:55:53.867281 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.867238 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a965e5de-653e-4302-8176-45e0229ed359-kube-api-access-7r44w" (OuterVolumeSpecName: "kube-api-access-7r44w") pod "a965e5de-653e-4302-8176-45e0229ed359" (UID: "a965e5de-653e-4302-8176-45e0229ed359"). InnerVolumeSpecName "kube-api-access-7r44w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:55:53.868356 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:53.868193 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wz2z6\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="a965e5de-653e-4302-8176-45e0229ed359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:53.870428 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:53.870351 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wz2z6\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="a965e5de-653e-4302-8176-45e0229ed359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:53.872509 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:53.872473 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rl2xq\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" Jul 21 21:55:53.873242 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.873220 2580 scope.go:117] "RemoveContainer" containerID="2342274d7ab865d53b6a0ce9fc3a91cf79063c49d33ad6fa5df1a69472cc4e05" Jul 21 21:55:53.873577 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:53.873551 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2342274d7ab865d53b6a0ce9fc3a91cf79063c49d33ad6fa5df1a69472cc4e05\": container with ID starting with 2342274d7ab865d53b6a0ce9fc3a91cf79063c49d33ad6fa5df1a69472cc4e05 not found: ID does not exist" containerID="2342274d7ab865d53b6a0ce9fc3a91cf79063c49d33ad6fa5df1a69472cc4e05" Jul 21 21:55:53.873661 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.873581 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2342274d7ab865d53b6a0ce9fc3a91cf79063c49d33ad6fa5df1a69472cc4e05"} err="failed to get container status \"2342274d7ab865d53b6a0ce9fc3a91cf79063c49d33ad6fa5df1a69472cc4e05\": rpc error: code = NotFound desc = could not find container \"2342274d7ab865d53b6a0ce9fc3a91cf79063c49d33ad6fa5df1a69472cc4e05\": container with ID starting with 2342274d7ab865d53b6a0ce9fc3a91cf79063c49d33ad6fa5df1a69472cc4e05 not found: ID does not exist" Jul 21 21:55:53.873661 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.873600 2580 scope.go:117] "RemoveContainer" containerID="9fc32be1a1246315b670e5440932254baff5ef8042de033958be38868b680fbe" Jul 21 21:55:53.874612 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:53.874592 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rch5j\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="9b7dcd4e-a265-487c-a47a-7fdc9753c119" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" Jul 21 21:55:53.881988 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.881968 2580 scope.go:117] "RemoveContainer" containerID="9fc32be1a1246315b670e5440932254baff5ef8042de033958be38868b680fbe" Jul 21 21:55:53.882325 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:53.882250 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc32be1a1246315b670e5440932254baff5ef8042de033958be38868b680fbe\": container with ID starting with 9fc32be1a1246315b670e5440932254baff5ef8042de033958be38868b680fbe not found: ID does not exist" containerID="9fc32be1a1246315b670e5440932254baff5ef8042de033958be38868b680fbe" Jul 21 21:55:53.882325 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.882275 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc32be1a1246315b670e5440932254baff5ef8042de033958be38868b680fbe"} err="failed to get container status \"9fc32be1a1246315b670e5440932254baff5ef8042de033958be38868b680fbe\": rpc error: code = NotFound desc = could not find container \"9fc32be1a1246315b670e5440932254baff5ef8042de033958be38868b680fbe\": container with ID starting with 9fc32be1a1246315b670e5440932254baff5ef8042de033958be38868b680fbe not found: ID does not exist" Jul 21 21:55:53.942955 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.942917 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr" Jul 21 21:55:53.948857 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.948828 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v" Jul 21 21:55:53.966737 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.966709 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r44w\" (UniqueName: \"kubernetes.io/projected/a965e5de-653e-4302-8176-45e0229ed359-kube-api-access-7r44w\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:55:53.966737 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:53.966736 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxnzw\" (UniqueName: \"kubernetes.io/projected/29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a-kube-api-access-bxnzw\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:55:54.115333 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:54.115307 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr"] Jul 21 21:55:54.118046 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:55:54.118019 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea578df_97e2_4f8e_acd7_8117b90f2175.slice/crio-8a64684ae1ef55feeb6b49bf4be4a9513f8882cf443a83b513d9d0c8c25689f5 WatchSource:0}: Error finding container 8a64684ae1ef55feeb6b49bf4be4a9513f8882cf443a83b513d9d0c8c25689f5: Status 404 returned error can't find the container with id 8a64684ae1ef55feeb6b49bf4be4a9513f8882cf443a83b513d9d0c8c25689f5 Jul 21 21:55:54.136407 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:54.136368 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v"] Jul 21 21:55:54.139816 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:55:54.139774 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ab49c4_8ca9_4aee_853c_73c55cb1b843.slice/crio-178b3e9c61a5b92f3745dae3c2974aa825d71d996d376fc7021b85cf78d28ed6 WatchSource:0}: Error finding container 178b3e9c61a5b92f3745dae3c2974aa825d71d996d376fc7021b85cf78d28ed6: Status 404 returned error can't find the container with id 178b3e9c61a5b92f3745dae3c2974aa825d71d996d376fc7021b85cf78d28ed6 Jul 21 21:55:54.199280 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:54.199239 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wz2z6\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="a965e5de-653e-4302-8176-45e0229ed359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:54.201921 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:54.201890 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rl2xq\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" Jul 21 21:55:54.204490 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:54.204457 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rch5j\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="9b7dcd4e-a265-487c-a47a-7fdc9753c119" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" Jul 21 21:55:54.206947 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:54.206911 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wz2z6\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="a965e5de-653e-4302-8176-45e0229ed359" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wz2z6" Jul 21 21:55:54.209344 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:54.209319 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rl2xq\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rl2xq" Jul 21 21:55:54.212124 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:54.212099 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rch5j\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="9b7dcd4e-a265-487c-a47a-7fdc9753c119" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" Jul 21 21:55:54.781632 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:54.781593 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a" path="/var/lib/kubelet/pods/29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a/volumes" Jul 21 21:55:54.781954 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:54.781942 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a965e5de-653e-4302-8176-45e0229ed359" path="/var/lib/kubelet/pods/a965e5de-653e-4302-8176-45e0229ed359/volumes" Jul 21 21:55:54.873730 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:54.873692 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr" event={"ID":"5ea578df-97e2-4f8e-acd7-8117b90f2175","Type":"ContainerStarted","Data":"3d31a25e55fa2833e2bdd34fd15565a064d64a408375f7634ea58f7839fcbd33"} Jul 21 21:55:54.874175 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:54.873760 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr" event={"ID":"5ea578df-97e2-4f8e-acd7-8117b90f2175","Type":"ContainerStarted","Data":"8a64684ae1ef55feeb6b49bf4be4a9513f8882cf443a83b513d9d0c8c25689f5"} Jul 21 21:55:54.874175 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:54.873906 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr" Jul 21 21:55:54.875643 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:54.875607 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v" event={"ID":"16ab49c4-8ca9-4aee-853c-73c55cb1b843","Type":"ContainerStarted","Data":"0824e009b0eb782a28e738257901f0293d406cf1cd56ba4b545d5cbbb97ada8c"} Jul 21 21:55:54.875843 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:54.875820 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v" event={"ID":"16ab49c4-8ca9-4aee-853c-73c55cb1b843","Type":"ContainerStarted","Data":"178b3e9c61a5b92f3745dae3c2974aa825d71d996d376fc7021b85cf78d28ed6"} Jul 21 21:55:54.875952 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:54.875861 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v" Jul 21 21:55:54.877287 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:54.877235 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" podUID="9b7dcd4e-a265-487c-a47a-7fdc9753c119" containerName="manager" containerID="cri-o://aed46b9a3f6c34810f6abeb2598cfadbfb72e1f617f781a9a779897cbcb164ac" gracePeriod=2 Jul 21 21:55:54.894632 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:54.894590 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr" podStartSLOduration=1.894574 podStartE2EDuration="1.894574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:55:54.893925847 +0000 UTC m=+432.657160108" watchObservedRunningTime="2026-07-21 21:55:54.894574 +0000 UTC m=+432.657808330" Jul 21 21:55:54.926817 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:54.926750 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v" podStartSLOduration=1.926729666 podStartE2EDuration="1.926729666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:55:54.924762167 +0000 UTC m=+432.687996430" watchObservedRunningTime="2026-07-21 21:55:54.926729666 +0000 UTC m=+432.689963929" Jul 21 21:55:55.126856 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:55.126830 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" Jul 21 21:55:55.129170 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:55.129143 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rch5j\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="9b7dcd4e-a265-487c-a47a-7fdc9753c119" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" Jul 21 21:55:55.279449 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:55.279419 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv5qv\" (UniqueName: \"kubernetes.io/projected/9b7dcd4e-a265-487c-a47a-7fdc9753c119-kube-api-access-zv5qv\") pod \"9b7dcd4e-a265-487c-a47a-7fdc9753c119\" (UID: \"9b7dcd4e-a265-487c-a47a-7fdc9753c119\") " Jul 21 21:55:55.279630 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:55.279564 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b7dcd4e-a265-487c-a47a-7fdc9753c119-extensions-socket-volume\") pod \"9b7dcd4e-a265-487c-a47a-7fdc9753c119\" (UID: \"9b7dcd4e-a265-487c-a47a-7fdc9753c119\") " Jul 21 21:55:55.279933 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:55.279909 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7dcd4e-a265-487c-a47a-7fdc9753c119-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9b7dcd4e-a265-487c-a47a-7fdc9753c119" (UID: "9b7dcd4e-a265-487c-a47a-7fdc9753c119"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:55:55.281660 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:55.281635 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7dcd4e-a265-487c-a47a-7fdc9753c119-kube-api-access-zv5qv" (OuterVolumeSpecName: "kube-api-access-zv5qv") pod "9b7dcd4e-a265-487c-a47a-7fdc9753c119" (UID: "9b7dcd4e-a265-487c-a47a-7fdc9753c119"). InnerVolumeSpecName "kube-api-access-zv5qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:55:55.381266 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:55.381194 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv5qv\" (UniqueName: \"kubernetes.io/projected/9b7dcd4e-a265-487c-a47a-7fdc9753c119-kube-api-access-zv5qv\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:55:55.381266 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:55.381219 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b7dcd4e-a265-487c-a47a-7fdc9753c119-extensions-socket-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:55:55.884594 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:55.884563 2580 generic.go:356] "Generic (PLEG): container finished" podID="9b7dcd4e-a265-487c-a47a-7fdc9753c119" containerID="aed46b9a3f6c34810f6abeb2598cfadbfb72e1f617f781a9a779897cbcb164ac" exitCode=0 Jul 21 21:55:55.885009 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:55.884604 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" Jul 21 21:55:55.885009 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:55.884660 2580 scope.go:117] "RemoveContainer" containerID="aed46b9a3f6c34810f6abeb2598cfadbfb72e1f617f781a9a779897cbcb164ac" Jul 21 21:55:55.887419 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:55.887390 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rch5j\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="9b7dcd4e-a265-487c-a47a-7fdc9753c119" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" Jul 21 21:55:55.893753 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:55.893738 2580 scope.go:117] "RemoveContainer" containerID="aed46b9a3f6c34810f6abeb2598cfadbfb72e1f617f781a9a779897cbcb164ac" Jul 21 21:55:55.894040 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:55.894019 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aed46b9a3f6c34810f6abeb2598cfadbfb72e1f617f781a9a779897cbcb164ac\": container with ID starting with aed46b9a3f6c34810f6abeb2598cfadbfb72e1f617f781a9a779897cbcb164ac not found: ID does not exist" containerID="aed46b9a3f6c34810f6abeb2598cfadbfb72e1f617f781a9a779897cbcb164ac" Jul 21 21:55:55.894104 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:55.894053 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aed46b9a3f6c34810f6abeb2598cfadbfb72e1f617f781a9a779897cbcb164ac"} err="failed to get container status \"aed46b9a3f6c34810f6abeb2598cfadbfb72e1f617f781a9a779897cbcb164ac\": rpc error: code = NotFound desc = could not find container \"aed46b9a3f6c34810f6abeb2598cfadbfb72e1f617f781a9a779897cbcb164ac\": container with ID starting with aed46b9a3f6c34810f6abeb2598cfadbfb72e1f617f781a9a779897cbcb164ac not found: ID does not exist" Jul 21 21:55:55.895872 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:55:55.895852 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rch5j\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="9b7dcd4e-a265-487c-a47a-7fdc9753c119" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rch5j" Jul 21 21:55:56.779348 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:55:56.779313 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b7dcd4e-a265-487c-a47a-7fdc9753c119" path="/var/lib/kubelet/pods/9b7dcd4e-a265-487c-a47a-7fdc9753c119/volumes" Jul 21 21:56:05.887722 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:05.887692 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v" Jul 21 21:56:05.888078 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:05.887761 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr" Jul 21 21:56:05.993287 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:05.993253 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v"] Jul 21 21:56:05.993480 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:05.993456 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v" podUID="16ab49c4-8ca9-4aee-853c-73c55cb1b843" containerName="manager" containerID="cri-o://0824e009b0eb782a28e738257901f0293d406cf1cd56ba4b545d5cbbb97ada8c" gracePeriod=10 Jul 21 21:56:06.226079 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.226046 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf"] Jul 21 21:56:06.226547 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.226529 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a" containerName="manager" Jul 21 21:56:06.226547 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.226549 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a" containerName="manager" Jul 21 21:56:06.226658 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.226617 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="29095fb9-0f81-45b2-a5c2-1cc0cfc7b71a" containerName="manager" Jul 21 21:56:06.229753 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.229738 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf" Jul 21 21:56:06.242538 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.242512 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf"] Jul 21 21:56:06.251662 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.251633 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v" Jul 21 21:56:06.280144 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.280119 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6jmj\" (UniqueName: \"kubernetes.io/projected/16ab49c4-8ca9-4aee-853c-73c55cb1b843-kube-api-access-n6jmj\") pod \"16ab49c4-8ca9-4aee-853c-73c55cb1b843\" (UID: \"16ab49c4-8ca9-4aee-853c-73c55cb1b843\") " Jul 21 21:56:06.280304 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.280159 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16ab49c4-8ca9-4aee-853c-73c55cb1b843-extensions-socket-volume\") pod \"16ab49c4-8ca9-4aee-853c-73c55cb1b843\" (UID: \"16ab49c4-8ca9-4aee-853c-73c55cb1b843\") " Jul 21 21:56:06.280438 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.280419 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aafea688-097a-4bf5-bd34-011da016e6c9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-f5lbf\" (UID: \"aafea688-097a-4bf5-bd34-011da016e6c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf" Jul 21 21:56:06.280499 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.280456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcq5g\" (UniqueName: \"kubernetes.io/projected/aafea688-097a-4bf5-bd34-011da016e6c9-kube-api-access-rcq5g\") pod \"kuadrant-operator-controller-manager-86878c769f-f5lbf\" (UID: \"aafea688-097a-4bf5-bd34-011da016e6c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf" Jul 21 21:56:06.280615 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.280596 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ab49c4-8ca9-4aee-853c-73c55cb1b843-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "16ab49c4-8ca9-4aee-853c-73c55cb1b843" (UID: "16ab49c4-8ca9-4aee-853c-73c55cb1b843"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:56:06.282338 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.282315 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ab49c4-8ca9-4aee-853c-73c55cb1b843-kube-api-access-n6jmj" (OuterVolumeSpecName: "kube-api-access-n6jmj") pod "16ab49c4-8ca9-4aee-853c-73c55cb1b843" (UID: "16ab49c4-8ca9-4aee-853c-73c55cb1b843"). InnerVolumeSpecName "kube-api-access-n6jmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:56:06.381425 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.381360 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aafea688-097a-4bf5-bd34-011da016e6c9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-f5lbf\" (UID: \"aafea688-097a-4bf5-bd34-011da016e6c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf" Jul 21 21:56:06.381425 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.381430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcq5g\" (UniqueName: \"kubernetes.io/projected/aafea688-097a-4bf5-bd34-011da016e6c9-kube-api-access-rcq5g\") pod \"kuadrant-operator-controller-manager-86878c769f-f5lbf\" (UID: \"aafea688-097a-4bf5-bd34-011da016e6c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf" Jul 21 21:56:06.381691 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.381515 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6jmj\" (UniqueName: \"kubernetes.io/projected/16ab49c4-8ca9-4aee-853c-73c55cb1b843-kube-api-access-n6jmj\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:56:06.381691 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.381530 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16ab49c4-8ca9-4aee-853c-73c55cb1b843-extensions-socket-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:56:06.387952 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.382191 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aafea688-097a-4bf5-bd34-011da016e6c9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-f5lbf\" (UID: \"aafea688-097a-4bf5-bd34-011da016e6c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf" Jul 21 21:56:06.391094 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.391066 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcq5g\" (UniqueName: \"kubernetes.io/projected/aafea688-097a-4bf5-bd34-011da016e6c9-kube-api-access-rcq5g\") pod \"kuadrant-operator-controller-manager-86878c769f-f5lbf\" (UID: \"aafea688-097a-4bf5-bd34-011da016e6c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf" Jul 21 21:56:06.557632 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.557590 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf" Jul 21 21:56:06.705082 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.705056 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf"] Jul 21 21:56:06.710292 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:56:06.709626 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafea688_097a_4bf5_bd34_011da016e6c9.slice/crio-b3b9afdd8d98c95070a36a31515df63fae106529caaf700259db4ee765692eff WatchSource:0}: Error finding container b3b9afdd8d98c95070a36a31515df63fae106529caaf700259db4ee765692eff: Status 404 returned error can't find the container with id b3b9afdd8d98c95070a36a31515df63fae106529caaf700259db4ee765692eff Jul 21 21:56:06.942060 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.941957 2580 generic.go:356] "Generic (PLEG): container finished" podID="16ab49c4-8ca9-4aee-853c-73c55cb1b843" containerID="0824e009b0eb782a28e738257901f0293d406cf1cd56ba4b545d5cbbb97ada8c" exitCode=0 Jul 21 21:56:06.942060 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.942015 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v" Jul 21 21:56:06.942060 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.942048 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v" event={"ID":"16ab49c4-8ca9-4aee-853c-73c55cb1b843","Type":"ContainerDied","Data":"0824e009b0eb782a28e738257901f0293d406cf1cd56ba4b545d5cbbb97ada8c"} Jul 21 21:56:06.942661 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.942110 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v" event={"ID":"16ab49c4-8ca9-4aee-853c-73c55cb1b843","Type":"ContainerDied","Data":"178b3e9c61a5b92f3745dae3c2974aa825d71d996d376fc7021b85cf78d28ed6"} Jul 21 21:56:06.942661 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.942144 2580 scope.go:117] "RemoveContainer" containerID="0824e009b0eb782a28e738257901f0293d406cf1cd56ba4b545d5cbbb97ada8c" Jul 21 21:56:06.943981 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.943953 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf" event={"ID":"aafea688-097a-4bf5-bd34-011da016e6c9","Type":"ContainerStarted","Data":"83f419730d3cfa270497f7bd9701035da0ec3bd1bcf2f5ba9ca441b98dd69de0"} Jul 21 21:56:06.944080 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.944012 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf" event={"ID":"aafea688-097a-4bf5-bd34-011da016e6c9","Type":"ContainerStarted","Data":"b3b9afdd8d98c95070a36a31515df63fae106529caaf700259db4ee765692eff"} Jul 21 21:56:06.944161 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.944127 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf" Jul 21 21:56:06.950917 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.950899 2580 scope.go:117] "RemoveContainer" containerID="0824e009b0eb782a28e738257901f0293d406cf1cd56ba4b545d5cbbb97ada8c" Jul 21 21:56:06.951205 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:56:06.951183 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0824e009b0eb782a28e738257901f0293d406cf1cd56ba4b545d5cbbb97ada8c\": container with ID starting with 0824e009b0eb782a28e738257901f0293d406cf1cd56ba4b545d5cbbb97ada8c not found: ID does not exist" containerID="0824e009b0eb782a28e738257901f0293d406cf1cd56ba4b545d5cbbb97ada8c" Jul 21 21:56:06.951275 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.951218 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0824e009b0eb782a28e738257901f0293d406cf1cd56ba4b545d5cbbb97ada8c"} err="failed to get container status \"0824e009b0eb782a28e738257901f0293d406cf1cd56ba4b545d5cbbb97ada8c\": rpc error: code = NotFound desc = could not find container \"0824e009b0eb782a28e738257901f0293d406cf1cd56ba4b545d5cbbb97ada8c\": container with ID starting with 0824e009b0eb782a28e738257901f0293d406cf1cd56ba4b545d5cbbb97ada8c not found: ID does not exist" Jul 21 21:56:06.972450 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.972404 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf" podStartSLOduration=0.972361716 podStartE2EDuration="972.361716ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:56:06.969790368 +0000 UTC m=+444.733024646" watchObservedRunningTime="2026-07-21 21:56:06.972361716 +0000 UTC m=+444.735595978" Jul 21 21:56:06.987700 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.987668 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v"] Jul 21 21:56:06.991684 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:06.991657 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9n29v"] Jul 21 21:56:08.780568 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:08.780534 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16ab49c4-8ca9-4aee-853c-73c55cb1b843" path="/var/lib/kubelet/pods/16ab49c4-8ca9-4aee-853c-73c55cb1b843/volumes" Jul 21 21:56:17.953217 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:17.953182 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf" Jul 21 21:56:18.006024 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:18.005990 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr"] Jul 21 21:56:18.006253 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:18.006227 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr" podUID="5ea578df-97e2-4f8e-acd7-8117b90f2175" containerName="manager" containerID="cri-o://3d31a25e55fa2833e2bdd34fd15565a064d64a408375f7634ea58f7839fcbd33" gracePeriod=10 Jul 21 21:56:18.248395 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:18.248352 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr" Jul 21 21:56:18.300944 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:18.300848 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ea578df-97e2-4f8e-acd7-8117b90f2175-extensions-socket-volume\") pod \"5ea578df-97e2-4f8e-acd7-8117b90f2175\" (UID: \"5ea578df-97e2-4f8e-acd7-8117b90f2175\") " Jul 21 21:56:18.301195 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:18.300952 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4vfz\" (UniqueName: \"kubernetes.io/projected/5ea578df-97e2-4f8e-acd7-8117b90f2175-kube-api-access-t4vfz\") pod \"5ea578df-97e2-4f8e-acd7-8117b90f2175\" (UID: \"5ea578df-97e2-4f8e-acd7-8117b90f2175\") " Jul 21 21:56:18.303908 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:18.303855 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea578df-97e2-4f8e-acd7-8117b90f2175-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5ea578df-97e2-4f8e-acd7-8117b90f2175" (UID: "5ea578df-97e2-4f8e-acd7-8117b90f2175"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:56:18.309019 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:18.308982 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea578df-97e2-4f8e-acd7-8117b90f2175-kube-api-access-t4vfz" (OuterVolumeSpecName: "kube-api-access-t4vfz") pod "5ea578df-97e2-4f8e-acd7-8117b90f2175" (UID: "5ea578df-97e2-4f8e-acd7-8117b90f2175"). InnerVolumeSpecName "kube-api-access-t4vfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:56:18.402799 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:18.402764 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4vfz\" (UniqueName: \"kubernetes.io/projected/5ea578df-97e2-4f8e-acd7-8117b90f2175-kube-api-access-t4vfz\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:56:18.402799 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:18.402792 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ea578df-97e2-4f8e-acd7-8117b90f2175-extensions-socket-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:56:19.007655 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:19.007616 2580 generic.go:356] "Generic (PLEG): container finished" podID="5ea578df-97e2-4f8e-acd7-8117b90f2175" containerID="3d31a25e55fa2833e2bdd34fd15565a064d64a408375f7634ea58f7839fcbd33" exitCode=0 Jul 21 21:56:19.008043 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:19.007685 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr" Jul 21 21:56:19.008043 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:19.007705 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr" event={"ID":"5ea578df-97e2-4f8e-acd7-8117b90f2175","Type":"ContainerDied","Data":"3d31a25e55fa2833e2bdd34fd15565a064d64a408375f7634ea58f7839fcbd33"} Jul 21 21:56:19.008043 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:19.007765 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr" event={"ID":"5ea578df-97e2-4f8e-acd7-8117b90f2175","Type":"ContainerDied","Data":"8a64684ae1ef55feeb6b49bf4be4a9513f8882cf443a83b513d9d0c8c25689f5"} Jul 21 21:56:19.008043 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:19.007786 2580 scope.go:117] "RemoveContainer" containerID="3d31a25e55fa2833e2bdd34fd15565a064d64a408375f7634ea58f7839fcbd33" Jul 21 21:56:19.016949 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:19.016931 2580 scope.go:117] "RemoveContainer" containerID="3d31a25e55fa2833e2bdd34fd15565a064d64a408375f7634ea58f7839fcbd33" Jul 21 21:56:19.017239 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:56:19.017218 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d31a25e55fa2833e2bdd34fd15565a064d64a408375f7634ea58f7839fcbd33\": container with ID starting with 3d31a25e55fa2833e2bdd34fd15565a064d64a408375f7634ea58f7839fcbd33 not found: ID does not exist" containerID="3d31a25e55fa2833e2bdd34fd15565a064d64a408375f7634ea58f7839fcbd33" Jul 21 21:56:19.017280 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:19.017248 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d31a25e55fa2833e2bdd34fd15565a064d64a408375f7634ea58f7839fcbd33"} err="failed to get container status \"3d31a25e55fa2833e2bdd34fd15565a064d64a408375f7634ea58f7839fcbd33\": rpc error: code = NotFound desc = could not find container \"3d31a25e55fa2833e2bdd34fd15565a064d64a408375f7634ea58f7839fcbd33\": container with ID starting with 3d31a25e55fa2833e2bdd34fd15565a064d64a408375f7634ea58f7839fcbd33 not found: ID does not exist" Jul 21 21:56:19.027258 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:19.027229 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr"] Jul 21 21:56:19.033051 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:19.033026 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hmqmr"] Jul 21 21:56:20.779682 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:20.779650 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ea578df-97e2-4f8e-acd7-8117b90f2175" path="/var/lib/kubelet/pods/5ea578df-97e2-4f8e-acd7-8117b90f2175/volumes" Jul 21 21:56:22.380093 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.380016 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6"] Jul 21 21:56:22.380438 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.380416 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16ab49c4-8ca9-4aee-853c-73c55cb1b843" containerName="manager" Jul 21 21:56:22.380438 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.380428 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="16ab49c4-8ca9-4aee-853c-73c55cb1b843" containerName="manager" Jul 21 21:56:22.380438 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.380438 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ea578df-97e2-4f8e-acd7-8117b90f2175" containerName="manager" Jul 21 21:56:22.380544 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.380443 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ea578df-97e2-4f8e-acd7-8117b90f2175" containerName="manager" Jul 21 21:56:22.380544 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.380507 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ea578df-97e2-4f8e-acd7-8117b90f2175" containerName="manager" Jul 21 21:56:22.380544 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.380518 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="16ab49c4-8ca9-4aee-853c-73c55cb1b843" containerName="manager" Jul 21 21:56:22.383735 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.383716 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.386288 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.386271 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lvvpx\"" Jul 21 21:56:22.395568 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.395548 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6"] Jul 21 21:56:22.436489 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.436453 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djctv\" (UniqueName: \"kubernetes.io/projected/de321670-e3b1-4976-9799-1bcb1e8d4823-kube-api-access-djctv\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.436661 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.436508 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de321670-e3b1-4976-9799-1bcb1e8d4823-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.436661 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.436534 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de321670-e3b1-4976-9799-1bcb1e8d4823-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.436661 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.436578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de321670-e3b1-4976-9799-1bcb1e8d4823-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.436661 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.436613 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de321670-e3b1-4976-9799-1bcb1e8d4823-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.436661 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.436645 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de321670-e3b1-4976-9799-1bcb1e8d4823-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.436821 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.436716 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de321670-e3b1-4976-9799-1bcb1e8d4823-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.436821 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.436747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de321670-e3b1-4976-9799-1bcb1e8d4823-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.436821 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.436776 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de321670-e3b1-4976-9799-1bcb1e8d4823-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.538020 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.537987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djctv\" (UniqueName: \"kubernetes.io/projected/de321670-e3b1-4976-9799-1bcb1e8d4823-kube-api-access-djctv\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.538218 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.538051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de321670-e3b1-4976-9799-1bcb1e8d4823-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.538218 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.538076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de321670-e3b1-4976-9799-1bcb1e8d4823-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.538218 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.538098 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de321670-e3b1-4976-9799-1bcb1e8d4823-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.538218 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.538119 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de321670-e3b1-4976-9799-1bcb1e8d4823-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.538218 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.538156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de321670-e3b1-4976-9799-1bcb1e8d4823-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.538218 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.538207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de321670-e3b1-4976-9799-1bcb1e8d4823-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.538544 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.538353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de321670-e3b1-4976-9799-1bcb1e8d4823-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.538544 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.538439 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de321670-e3b1-4976-9799-1bcb1e8d4823-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.538663 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.538635 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de321670-e3b1-4976-9799-1bcb1e8d4823-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.538718 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.538671 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de321670-e3b1-4976-9799-1bcb1e8d4823-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.538776 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.538743 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de321670-e3b1-4976-9799-1bcb1e8d4823-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.538776 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.538752 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de321670-e3b1-4976-9799-1bcb1e8d4823-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.539120 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.539098 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de321670-e3b1-4976-9799-1bcb1e8d4823-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.540608 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.540589 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de321670-e3b1-4976-9799-1bcb1e8d4823-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.540807 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.540790 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de321670-e3b1-4976-9799-1bcb1e8d4823-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.546645 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.546625 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de321670-e3b1-4976-9799-1bcb1e8d4823-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.546819 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.546800 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djctv\" (UniqueName: \"kubernetes.io/projected/de321670-e3b1-4976-9799-1bcb1e8d4823-kube-api-access-djctv\") pod \"maas-default-gateway-openshift-default-56654dcf7-bwxc6\" (UID: \"de321670-e3b1-4976-9799-1bcb1e8d4823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.694563 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.694469 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:22.847105 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.842368 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6"] Jul 21 21:56:22.847105 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.845985 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 21:56:22.847105 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.846080 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 21:56:22.847105 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:22.846136 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 21:56:23.030715 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:23.030673 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" event={"ID":"de321670-e3b1-4976-9799-1bcb1e8d4823","Type":"ContainerStarted","Data":"430e259afad8c96efc9756eed2202f3b673fbbaf6b6e0eb74b09e88c93b40b7a"} Jul 21 21:56:23.030914 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:23.030737 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" event={"ID":"de321670-e3b1-4976-9799-1bcb1e8d4823","Type":"ContainerStarted","Data":"6c5ca16fc1b33145347d89bb74097f70f371991a7b264568ae972b5c577c45de"} Jul 21 21:56:23.056428 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:23.056351 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" podStartSLOduration=1.0563339 podStartE2EDuration="1.0563339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:56:23.054145664 +0000 UTC m=+460.817379981" watchObservedRunningTime="2026-07-21 21:56:23.0563339 +0000 UTC m=+460.819568160" Jul 21 21:56:23.695295 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:23.695264 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:23.699683 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:23.699660 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:24.036220 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:24.036187 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:24.037085 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:24.037062 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:56:37.252600 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:37.252565 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-glcz2"] Jul 21 21:56:37.255204 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:37.255189 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-glcz2" Jul 21 21:56:37.257867 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:37.257839 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 21 21:56:37.257993 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:37.257943 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cpkrr\"" Jul 21 21:56:37.264132 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:37.264114 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-glcz2"] Jul 21 21:56:37.345601 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:37.345571 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-glcz2"] Jul 21 21:56:37.367910 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:37.367881 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af2fd287-b2a6-414f-88a4-cb7984601e7b-config-file\") pod \"limitador-limitador-64c94fd64f-glcz2\" (UID: \"af2fd287-b2a6-414f-88a4-cb7984601e7b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-glcz2" Jul 21 21:56:37.368039 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:37.367923 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgksq\" (UniqueName: \"kubernetes.io/projected/af2fd287-b2a6-414f-88a4-cb7984601e7b-kube-api-access-bgksq\") pod \"limitador-limitador-64c94fd64f-glcz2\" (UID: \"af2fd287-b2a6-414f-88a4-cb7984601e7b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-glcz2" Jul 21 21:56:37.468854 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:37.468817 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af2fd287-b2a6-414f-88a4-cb7984601e7b-config-file\") pod \"limitador-limitador-64c94fd64f-glcz2\" (UID: \"af2fd287-b2a6-414f-88a4-cb7984601e7b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-glcz2" Jul 21 21:56:37.469037 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:37.468876 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgksq\" (UniqueName: \"kubernetes.io/projected/af2fd287-b2a6-414f-88a4-cb7984601e7b-kube-api-access-bgksq\") pod \"limitador-limitador-64c94fd64f-glcz2\" (UID: \"af2fd287-b2a6-414f-88a4-cb7984601e7b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-glcz2" Jul 21 21:56:37.469555 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:37.469530 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af2fd287-b2a6-414f-88a4-cb7984601e7b-config-file\") pod \"limitador-limitador-64c94fd64f-glcz2\" (UID: \"af2fd287-b2a6-414f-88a4-cb7984601e7b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-glcz2" Jul 21 21:56:37.477781 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:37.477755 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgksq\" (UniqueName: \"kubernetes.io/projected/af2fd287-b2a6-414f-88a4-cb7984601e7b-kube-api-access-bgksq\") pod \"limitador-limitador-64c94fd64f-glcz2\" (UID: \"af2fd287-b2a6-414f-88a4-cb7984601e7b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-glcz2" Jul 21 21:56:37.566152 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:37.566072 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-glcz2" Jul 21 21:56:37.702913 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:37.702889 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-glcz2"] Jul 21 21:56:37.705483 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:56:37.705443 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2fd287_b2a6_414f_88a4_cb7984601e7b.slice/crio-5da0a6508dcf417d9ec1283e5710bf2c13e3ba818307c9685adfe74d330a3ab6 WatchSource:0}: Error finding container 5da0a6508dcf417d9ec1283e5710bf2c13e3ba818307c9685adfe74d330a3ab6: Status 404 returned error can't find the container with id 5da0a6508dcf417d9ec1283e5710bf2c13e3ba818307c9685adfe74d330a3ab6 Jul 21 21:56:38.106111 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:38.106073 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-glcz2" event={"ID":"af2fd287-b2a6-414f-88a4-cb7984601e7b","Type":"ContainerStarted","Data":"5da0a6508dcf417d9ec1283e5710bf2c13e3ba818307c9685adfe74d330a3ab6"} Jul 21 21:56:38.192948 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:38.192914 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-gg6fv"] Jul 21 21:56:38.195868 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:38.195851 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gg6fv" Jul 21 21:56:38.198468 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:38.198450 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t5swz\"" Jul 21 21:56:38.203800 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:38.203782 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gg6fv"] Jul 21 21:56:38.277219 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:38.277188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22f65\" (UniqueName: \"kubernetes.io/projected/15faee94-8483-4e7e-89df-543bb13771af-kube-api-access-22f65\") pod \"authorino-75d556bd77-gg6fv\" (UID: \"15faee94-8483-4e7e-89df-543bb13771af\") " pod="kuadrant-system/authorino-75d556bd77-gg6fv" Jul 21 21:56:38.378670 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:38.378609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22f65\" (UniqueName: \"kubernetes.io/projected/15faee94-8483-4e7e-89df-543bb13771af-kube-api-access-22f65\") pod \"authorino-75d556bd77-gg6fv\" (UID: \"15faee94-8483-4e7e-89df-543bb13771af\") " pod="kuadrant-system/authorino-75d556bd77-gg6fv" Jul 21 21:56:38.386971 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:38.386948 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22f65\" (UniqueName: \"kubernetes.io/projected/15faee94-8483-4e7e-89df-543bb13771af-kube-api-access-22f65\") pod \"authorino-75d556bd77-gg6fv\" (UID: \"15faee94-8483-4e7e-89df-543bb13771af\") " pod="kuadrant-system/authorino-75d556bd77-gg6fv" Jul 21 21:56:38.505992 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:38.505960 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gg6fv" Jul 21 21:56:38.689884 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:38.689770 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gg6fv"] Jul 21 21:56:38.692697 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:56:38.692668 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15faee94_8483_4e7e_89df_543bb13771af.slice/crio-b92afe957f7c0b87edb60523ee9a88369d1da26c4b53ff22909d7e859922771a WatchSource:0}: Error finding container b92afe957f7c0b87edb60523ee9a88369d1da26c4b53ff22909d7e859922771a: Status 404 returned error can't find the container with id b92afe957f7c0b87edb60523ee9a88369d1da26c4b53ff22909d7e859922771a Jul 21 21:56:39.113411 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:39.113348 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gg6fv" event={"ID":"15faee94-8483-4e7e-89df-543bb13771af","Type":"ContainerStarted","Data":"b92afe957f7c0b87edb60523ee9a88369d1da26c4b53ff22909d7e859922771a"} Jul 21 21:56:41.128284 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:41.128229 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-glcz2" event={"ID":"af2fd287-b2a6-414f-88a4-cb7984601e7b","Type":"ContainerStarted","Data":"38b2b8d6f55eb48039a74075aff34fdba55cd6eaf1b2e6835af156fbfb40607d"} Jul 21 21:56:41.128760 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:41.128424 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-glcz2" Jul 21 21:56:41.147124 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:41.147080 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-glcz2" podStartSLOduration=1.296522487 podStartE2EDuration="4.14706518s" totalImagesPullingTime="2.850542693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:56:41.145922114 +0000 UTC m=+478.909156376" watchObservedRunningTime="2026-07-21 21:56:41.14706518 +0000 UTC m=+478.910299440" Jul 21 21:56:43.141655 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:43.141623 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gg6fv" event={"ID":"15faee94-8483-4e7e-89df-543bb13771af","Type":"ContainerStarted","Data":"c583262e3f1bbe5ec0b10b7a0631ccf8d6fe678ba2af4fc720e12a343434b6a0"} Jul 21 21:56:43.157785 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:43.157736 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-gg6fv" podStartSLOduration=1.189414674 podStartE2EDuration="5.157716856s" totalImagesPullingTime="3.968302182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:56:43.156683493 +0000 UTC m=+480.919917755" watchObservedRunningTime="2026-07-21 21:56:43.157716856 +0000 UTC m=+480.920951117" Jul 21 21:56:52.136755 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:52.136721 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-glcz2" Jul 21 21:56:53.876627 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:53.876596 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-glcz2"] Jul 21 21:56:53.877063 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:53.876786 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-glcz2" podUID="af2fd287-b2a6-414f-88a4-cb7984601e7b" containerName="limitador" containerID="cri-o://38b2b8d6f55eb48039a74075aff34fdba55cd6eaf1b2e6835af156fbfb40607d" gracePeriod=30 Jul 21 21:56:54.419217 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:54.419194 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-glcz2" Jul 21 21:56:54.423006 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:54.422990 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgksq\" (UniqueName: \"kubernetes.io/projected/af2fd287-b2a6-414f-88a4-cb7984601e7b-kube-api-access-bgksq\") pod \"af2fd287-b2a6-414f-88a4-cb7984601e7b\" (UID: \"af2fd287-b2a6-414f-88a4-cb7984601e7b\") " Jul 21 21:56:54.423080 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:54.423065 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af2fd287-b2a6-414f-88a4-cb7984601e7b-config-file\") pod \"af2fd287-b2a6-414f-88a4-cb7984601e7b\" (UID: \"af2fd287-b2a6-414f-88a4-cb7984601e7b\") " Jul 21 21:56:54.423411 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:54.423392 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af2fd287-b2a6-414f-88a4-cb7984601e7b-config-file" (OuterVolumeSpecName: "config-file") pod "af2fd287-b2a6-414f-88a4-cb7984601e7b" (UID: "af2fd287-b2a6-414f-88a4-cb7984601e7b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:56:54.425065 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:54.425043 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2fd287-b2a6-414f-88a4-cb7984601e7b-kube-api-access-bgksq" (OuterVolumeSpecName: "kube-api-access-bgksq") pod "af2fd287-b2a6-414f-88a4-cb7984601e7b" (UID: "af2fd287-b2a6-414f-88a4-cb7984601e7b"). InnerVolumeSpecName "kube-api-access-bgksq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:56:54.524256 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:54.524228 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgksq\" (UniqueName: \"kubernetes.io/projected/af2fd287-b2a6-414f-88a4-cb7984601e7b-kube-api-access-bgksq\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:56:54.524256 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:54.524256 2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af2fd287-b2a6-414f-88a4-cb7984601e7b-config-file\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:56:55.199747 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:55.199660 2580 generic.go:356] "Generic (PLEG): container finished" podID="af2fd287-b2a6-414f-88a4-cb7984601e7b" containerID="38b2b8d6f55eb48039a74075aff34fdba55cd6eaf1b2e6835af156fbfb40607d" exitCode=0 Jul 21 21:56:55.199747 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:55.199706 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-glcz2" Jul 21 21:56:55.199747 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:55.199717 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-glcz2" event={"ID":"af2fd287-b2a6-414f-88a4-cb7984601e7b","Type":"ContainerDied","Data":"38b2b8d6f55eb48039a74075aff34fdba55cd6eaf1b2e6835af156fbfb40607d"} Jul 21 21:56:55.200224 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:55.199765 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-glcz2" event={"ID":"af2fd287-b2a6-414f-88a4-cb7984601e7b","Type":"ContainerDied","Data":"5da0a6508dcf417d9ec1283e5710bf2c13e3ba818307c9685adfe74d330a3ab6"} Jul 21 21:56:55.200224 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:55.199789 2580 scope.go:117] "RemoveContainer" containerID="38b2b8d6f55eb48039a74075aff34fdba55cd6eaf1b2e6835af156fbfb40607d" Jul 21 21:56:55.208307 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:55.208288 2580 scope.go:117] "RemoveContainer" containerID="38b2b8d6f55eb48039a74075aff34fdba55cd6eaf1b2e6835af156fbfb40607d" Jul 21 21:56:55.208638 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:56:55.208615 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b2b8d6f55eb48039a74075aff34fdba55cd6eaf1b2e6835af156fbfb40607d\": container with ID starting with 38b2b8d6f55eb48039a74075aff34fdba55cd6eaf1b2e6835af156fbfb40607d not found: ID does not exist" containerID="38b2b8d6f55eb48039a74075aff34fdba55cd6eaf1b2e6835af156fbfb40607d" Jul 21 21:56:55.208742 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:55.208645 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b2b8d6f55eb48039a74075aff34fdba55cd6eaf1b2e6835af156fbfb40607d"} err="failed to get container status \"38b2b8d6f55eb48039a74075aff34fdba55cd6eaf1b2e6835af156fbfb40607d\": rpc error: code = NotFound desc = could not find container \"38b2b8d6f55eb48039a74075aff34fdba55cd6eaf1b2e6835af156fbfb40607d\": container with ID starting with 38b2b8d6f55eb48039a74075aff34fdba55cd6eaf1b2e6835af156fbfb40607d not found: ID does not exist" Jul 21 21:56:55.219145 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:55.219122 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-glcz2"] Jul 21 21:56:55.223358 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:55.223339 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-glcz2"] Jul 21 21:56:56.050899 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.050869 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-m7ls7"] Jul 21 21:56:56.054504 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.052188 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af2fd287-b2a6-414f-88a4-cb7984601e7b" containerName="limitador" Jul 21 21:56:56.054504 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.052227 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="af2fd287-b2a6-414f-88a4-cb7984601e7b" containerName="limitador" Jul 21 21:56:56.054504 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.052461 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="af2fd287-b2a6-414f-88a4-cb7984601e7b" containerName="limitador" Jul 21 21:56:56.058103 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.058075 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-m7ls7" Jul 21 21:56:56.061155 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.061129 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 21:56:56.061274 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.061256 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 21 21:56:56.062328 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.062309 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 21 21:56:56.062328 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.062319 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-7bvsd\"" Jul 21 21:56:56.066139 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.066114 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-m7ls7"] Jul 21 21:56:56.142209 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.142177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhvk2\" (UniqueName: \"kubernetes.io/projected/9b2239bf-33cf-4f63-a40d-92394cb9fde8-kube-api-access-nhvk2\") pod \"postgres-69fb4594cf-m7ls7\" (UID: \"9b2239bf-33cf-4f63-a40d-92394cb9fde8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-m7ls7" Jul 21 21:56:56.142336 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.142306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9b2239bf-33cf-4f63-a40d-92394cb9fde8-data\") pod \"postgres-69fb4594cf-m7ls7\" (UID: \"9b2239bf-33cf-4f63-a40d-92394cb9fde8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-m7ls7" Jul 21 21:56:56.243277 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.243243 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhvk2\" (UniqueName: \"kubernetes.io/projected/9b2239bf-33cf-4f63-a40d-92394cb9fde8-kube-api-access-nhvk2\") pod \"postgres-69fb4594cf-m7ls7\" (UID: \"9b2239bf-33cf-4f63-a40d-92394cb9fde8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-m7ls7" Jul 21 21:56:56.243700 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.243369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9b2239bf-33cf-4f63-a40d-92394cb9fde8-data\") pod \"postgres-69fb4594cf-m7ls7\" (UID: \"9b2239bf-33cf-4f63-a40d-92394cb9fde8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-m7ls7" Jul 21 21:56:56.243746 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.243698 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9b2239bf-33cf-4f63-a40d-92394cb9fde8-data\") pod \"postgres-69fb4594cf-m7ls7\" (UID: \"9b2239bf-33cf-4f63-a40d-92394cb9fde8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-m7ls7" Jul 21 21:56:56.252291 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.252273 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhvk2\" (UniqueName: \"kubernetes.io/projected/9b2239bf-33cf-4f63-a40d-92394cb9fde8-kube-api-access-nhvk2\") pod \"postgres-69fb4594cf-m7ls7\" (UID: \"9b2239bf-33cf-4f63-a40d-92394cb9fde8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-m7ls7" Jul 21 21:56:56.370779 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.370713 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-m7ls7" Jul 21 21:56:56.507514 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.507437 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-m7ls7"] Jul 21 21:56:56.509931 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:56:56.509906 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2239bf_33cf_4f63_a40d_92394cb9fde8.slice/crio-2036bed90fe397d72293cfe61b37f426bffd77637fd6fcaaa9f37f0972a2453b WatchSource:0}: Error finding container 2036bed90fe397d72293cfe61b37f426bffd77637fd6fcaaa9f37f0972a2453b: Status 404 returned error can't find the container with id 2036bed90fe397d72293cfe61b37f426bffd77637fd6fcaaa9f37f0972a2453b Jul 21 21:56:56.779294 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:56.779262 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af2fd287-b2a6-414f-88a4-cb7984601e7b" path="/var/lib/kubelet/pods/af2fd287-b2a6-414f-88a4-cb7984601e7b/volumes" Jul 21 21:56:57.213739 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:56:57.213617 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-m7ls7" event={"ID":"9b2239bf-33cf-4f63-a40d-92394cb9fde8","Type":"ContainerStarted","Data":"2036bed90fe397d72293cfe61b37f426bffd77637fd6fcaaa9f37f0972a2453b"} Jul 21 21:57:02.244018 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:02.243982 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-m7ls7" event={"ID":"9b2239bf-33cf-4f63-a40d-92394cb9fde8","Type":"ContainerStarted","Data":"843aef6af722928b474e1d7f2f2fb347b8ffe7e027886188a7099208b7203090"} Jul 21 21:57:02.244441 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:02.244161 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-m7ls7" Jul 21 21:57:02.262843 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:02.262669 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-m7ls7" podStartSLOduration=1.54618795 podStartE2EDuration="6.262650479s" totalImagesPullingTime="4.716462529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:57:02.262207659 +0000 UTC m=+500.025441924" watchObservedRunningTime="2026-07-21 21:57:02.262650479 +0000 UTC m=+500.025884741" Jul 21 21:57:08.281946 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:08.281918 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-m7ls7" Jul 21 21:57:08.851128 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:08.851100 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pnqt6"] Jul 21 21:57:08.857089 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:08.857067 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pnqt6" Jul 21 21:57:08.862951 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:08.862923 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pnqt6"] Jul 21 21:57:08.966877 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:08.966846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw6zw\" (UniqueName: \"kubernetes.io/projected/abb35aa8-d003-42b1-8c94-a17afa96d114-kube-api-access-fw6zw\") pod \"authorino-5d4b67c446-pnqt6\" (UID: \"abb35aa8-d003-42b1-8c94-a17afa96d114\") " pod="kuadrant-system/authorino-5d4b67c446-pnqt6" Jul 21 21:57:09.067813 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.067778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw6zw\" (UniqueName: \"kubernetes.io/projected/abb35aa8-d003-42b1-8c94-a17afa96d114-kube-api-access-fw6zw\") pod \"authorino-5d4b67c446-pnqt6\" (UID: \"abb35aa8-d003-42b1-8c94-a17afa96d114\") " pod="kuadrant-system/authorino-5d4b67c446-pnqt6" Jul 21 21:57:09.078602 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.078572 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw6zw\" (UniqueName: \"kubernetes.io/projected/abb35aa8-d003-42b1-8c94-a17afa96d114-kube-api-access-fw6zw\") pod \"authorino-5d4b67c446-pnqt6\" (UID: \"abb35aa8-d003-42b1-8c94-a17afa96d114\") " pod="kuadrant-system/authorino-5d4b67c446-pnqt6" Jul 21 21:57:09.087728 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.087703 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pnqt6"] Jul 21 21:57:09.087968 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.087957 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pnqt6" Jul 21 21:57:09.118587 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.118530 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f5d5dbc89-fpz44"] Jul 21 21:57:09.122756 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.122736 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f5d5dbc89-fpz44" Jul 21 21:57:09.133543 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.133514 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f5d5dbc89-fpz44"] Jul 21 21:57:09.212675 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.212645 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f5d5dbc89-fpz44"] Jul 21 21:57:09.212968 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:57:09.212948 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-nrlrm], unattached volumes=[], failed to process volumes=[kube-api-access-nrlrm]: context canceled" pod="kuadrant-system/authorino-7f5d5dbc89-fpz44" podUID="51000fea-664c-4c79-95f7-b69a7a9d73dd" Jul 21 21:57:09.228494 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.228475 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pnqt6"] Jul 21 21:57:09.231026 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:57:09.231001 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabb35aa8_d003_42b1_8c94_a17afa96d114.slice/crio-07090cb8ce84147364574e907ef6a2661c5dfed451d79382403027db4ff59163 WatchSource:0}: Error finding container 07090cb8ce84147364574e907ef6a2661c5dfed451d79382403027db4ff59163: Status 404 returned error can't find the container with id 07090cb8ce84147364574e907ef6a2661c5dfed451d79382403027db4ff59163 Jul 21 21:57:09.251309 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.251288 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5569887bcc-swrx8"] Jul 21 21:57:09.255135 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.255119 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5569887bcc-swrx8" Jul 21 21:57:09.258082 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.258063 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 21 21:57:09.264657 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.264638 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5569887bcc-swrx8"] Jul 21 21:57:09.269718 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.269698 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrlrm\" (UniqueName: \"kubernetes.io/projected/51000fea-664c-4c79-95f7-b69a7a9d73dd-kube-api-access-nrlrm\") pod \"authorino-7f5d5dbc89-fpz44\" (UID: \"51000fea-664c-4c79-95f7-b69a7a9d73dd\") " pod="kuadrant-system/authorino-7f5d5dbc89-fpz44" Jul 21 21:57:09.280966 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.280949 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f5d5dbc89-fpz44" Jul 21 21:57:09.281093 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.280973 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pnqt6" event={"ID":"abb35aa8-d003-42b1-8c94-a17afa96d114","Type":"ContainerStarted","Data":"07090cb8ce84147364574e907ef6a2661c5dfed451d79382403027db4ff59163"} Jul 21 21:57:09.287730 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.287709 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f5d5dbc89-fpz44" Jul 21 21:57:09.370827 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.370770 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74czh\" (UniqueName: \"kubernetes.io/projected/cc169b0d-34ca-4a21-84c3-16fe6de2174a-kube-api-access-74czh\") pod \"authorino-5569887bcc-swrx8\" (UID: \"cc169b0d-34ca-4a21-84c3-16fe6de2174a\") " pod="kuadrant-system/authorino-5569887bcc-swrx8" Jul 21 21:57:09.370827 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.370811 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc169b0d-34ca-4a21-84c3-16fe6de2174a-tls-cert\") pod \"authorino-5569887bcc-swrx8\" (UID: \"cc169b0d-34ca-4a21-84c3-16fe6de2174a\") " pod="kuadrant-system/authorino-5569887bcc-swrx8" Jul 21 21:57:09.370968 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.370876 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrlrm\" (UniqueName: \"kubernetes.io/projected/51000fea-664c-4c79-95f7-b69a7a9d73dd-kube-api-access-nrlrm\") pod \"authorino-7f5d5dbc89-fpz44\" (UID: \"51000fea-664c-4c79-95f7-b69a7a9d73dd\") " pod="kuadrant-system/authorino-7f5d5dbc89-fpz44" Jul 21 21:57:09.380113 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.380092 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrlrm\" (UniqueName: \"kubernetes.io/projected/51000fea-664c-4c79-95f7-b69a7a9d73dd-kube-api-access-nrlrm\") pod \"authorino-7f5d5dbc89-fpz44\" (UID: \"51000fea-664c-4c79-95f7-b69a7a9d73dd\") " pod="kuadrant-system/authorino-7f5d5dbc89-fpz44" Jul 21 21:57:09.472307 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.472283 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrlrm\" (UniqueName: \"kubernetes.io/projected/51000fea-664c-4c79-95f7-b69a7a9d73dd-kube-api-access-nrlrm\") pod \"51000fea-664c-4c79-95f7-b69a7a9d73dd\" (UID: \"51000fea-664c-4c79-95f7-b69a7a9d73dd\") " Jul 21 21:57:09.472497 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.472481 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74czh\" (UniqueName: \"kubernetes.io/projected/cc169b0d-34ca-4a21-84c3-16fe6de2174a-kube-api-access-74czh\") pod \"authorino-5569887bcc-swrx8\" (UID: \"cc169b0d-34ca-4a21-84c3-16fe6de2174a\") " pod="kuadrant-system/authorino-5569887bcc-swrx8" Jul 21 21:57:09.472544 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.472519 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc169b0d-34ca-4a21-84c3-16fe6de2174a-tls-cert\") pod \"authorino-5569887bcc-swrx8\" (UID: \"cc169b0d-34ca-4a21-84c3-16fe6de2174a\") " pod="kuadrant-system/authorino-5569887bcc-swrx8" Jul 21 21:57:09.474351 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.474321 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51000fea-664c-4c79-95f7-b69a7a9d73dd-kube-api-access-nrlrm" (OuterVolumeSpecName: "kube-api-access-nrlrm") pod "51000fea-664c-4c79-95f7-b69a7a9d73dd" (UID: "51000fea-664c-4c79-95f7-b69a7a9d73dd"). InnerVolumeSpecName "kube-api-access-nrlrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:57:09.474817 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.474802 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc169b0d-34ca-4a21-84c3-16fe6de2174a-tls-cert\") pod \"authorino-5569887bcc-swrx8\" (UID: \"cc169b0d-34ca-4a21-84c3-16fe6de2174a\") " pod="kuadrant-system/authorino-5569887bcc-swrx8" Jul 21 21:57:09.480898 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.480881 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74czh\" (UniqueName: \"kubernetes.io/projected/cc169b0d-34ca-4a21-84c3-16fe6de2174a-kube-api-access-74czh\") pod \"authorino-5569887bcc-swrx8\" (UID: \"cc169b0d-34ca-4a21-84c3-16fe6de2174a\") " pod="kuadrant-system/authorino-5569887bcc-swrx8" Jul 21 21:57:09.565282 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.565259 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5569887bcc-swrx8" Jul 21 21:57:09.573767 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.573750 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrlrm\" (UniqueName: \"kubernetes.io/projected/51000fea-664c-4c79-95f7-b69a7a9d73dd-kube-api-access-nrlrm\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:57:09.695298 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:09.695274 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5569887bcc-swrx8"] Jul 21 21:57:09.697934 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:57:09.697901 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc169b0d_34ca_4a21_84c3_16fe6de2174a.slice/crio-1922fe9d67215d76613b21bc015a8e27e1735ecddb7c77d6a61209d015ef77a2 WatchSource:0}: Error finding container 1922fe9d67215d76613b21bc015a8e27e1735ecddb7c77d6a61209d015ef77a2: Status 404 returned error can't find the container with id 1922fe9d67215d76613b21bc015a8e27e1735ecddb7c77d6a61209d015ef77a2 Jul 21 21:57:10.286942 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.286898 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5569887bcc-swrx8" event={"ID":"cc169b0d-34ca-4a21-84c3-16fe6de2174a","Type":"ContainerStarted","Data":"2197161167d359214197b909ae02f0372348bcf31aa5a49c0fd915b1b889aba8"} Jul 21 21:57:10.287123 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.286952 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5569887bcc-swrx8" event={"ID":"cc169b0d-34ca-4a21-84c3-16fe6de2174a","Type":"ContainerStarted","Data":"1922fe9d67215d76613b21bc015a8e27e1735ecddb7c77d6a61209d015ef77a2"} Jul 21 21:57:10.288434 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.288404 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pnqt6" event={"ID":"abb35aa8-d003-42b1-8c94-a17afa96d114","Type":"ContainerStarted","Data":"868670ae904105e9c72ad73e604f3d0b8c4a7b3112d488a91dd3d272382999e4"} Jul 21 21:57:10.288776 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.288419 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f5d5dbc89-fpz44" Jul 21 21:57:10.288776 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.288465 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-pnqt6" podUID="abb35aa8-d003-42b1-8c94-a17afa96d114" containerName="authorino" containerID="cri-o://868670ae904105e9c72ad73e604f3d0b8c4a7b3112d488a91dd3d272382999e4" gracePeriod=30 Jul 21 21:57:10.306758 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.306721 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5569887bcc-swrx8" podStartSLOduration=1.026150095 podStartE2EDuration="1.306706746s" totalImagesPullingTime="280.556651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:57:10.304494642 +0000 UTC m=+508.067728905" watchObservedRunningTime="2026-07-21 21:57:10.306706746 +0000 UTC m=+508.069941006" Jul 21 21:57:10.322938 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.322887 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-pnqt6" podStartSLOduration=1.861012144 podStartE2EDuration="2.322868709s" totalImagesPullingTime="461.856565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:57:10.322282081 +0000 UTC m=+508.085516355" watchObservedRunningTime="2026-07-21 21:57:10.322868709 +0000 UTC m=+508.086102971" Jul 21 21:57:10.341705 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.341680 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gg6fv"] Jul 21 21:57:10.341860 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.341841 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-gg6fv" podUID="15faee94-8483-4e7e-89df-543bb13771af" containerName="authorino" containerID="cri-o://c583262e3f1bbe5ec0b10b7a0631ccf8d6fe678ba2af4fc720e12a343434b6a0" gracePeriod=30 Jul 21 21:57:10.348317 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.348291 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f5d5dbc89-fpz44"] Jul 21 21:57:10.352117 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.352094 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7f5d5dbc89-fpz44"] Jul 21 21:57:10.612058 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.612030 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pnqt6" Jul 21 21:57:10.614960 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.614941 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gg6fv" Jul 21 21:57:10.684523 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.684497 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22f65\" (UniqueName: \"kubernetes.io/projected/15faee94-8483-4e7e-89df-543bb13771af-kube-api-access-22f65\") pod \"15faee94-8483-4e7e-89df-543bb13771af\" (UID: \"15faee94-8483-4e7e-89df-543bb13771af\") " Jul 21 21:57:10.684691 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.684553 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw6zw\" (UniqueName: \"kubernetes.io/projected/abb35aa8-d003-42b1-8c94-a17afa96d114-kube-api-access-fw6zw\") pod \"abb35aa8-d003-42b1-8c94-a17afa96d114\" (UID: \"abb35aa8-d003-42b1-8c94-a17afa96d114\") " Jul 21 21:57:10.686478 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.686444 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15faee94-8483-4e7e-89df-543bb13771af-kube-api-access-22f65" (OuterVolumeSpecName: "kube-api-access-22f65") pod "15faee94-8483-4e7e-89df-543bb13771af" (UID: "15faee94-8483-4e7e-89df-543bb13771af"). InnerVolumeSpecName "kube-api-access-22f65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:57:10.686597 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.686569 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb35aa8-d003-42b1-8c94-a17afa96d114-kube-api-access-fw6zw" (OuterVolumeSpecName: "kube-api-access-fw6zw") pod "abb35aa8-d003-42b1-8c94-a17afa96d114" (UID: "abb35aa8-d003-42b1-8c94-a17afa96d114"). InnerVolumeSpecName "kube-api-access-fw6zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:57:10.779765 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.779735 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51000fea-664c-4c79-95f7-b69a7a9d73dd" path="/var/lib/kubelet/pods/51000fea-664c-4c79-95f7-b69a7a9d73dd/volumes" Jul 21 21:57:10.787892 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.787863 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22f65\" (UniqueName: \"kubernetes.io/projected/15faee94-8483-4e7e-89df-543bb13771af-kube-api-access-22f65\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:57:10.788028 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:10.787895 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw6zw\" (UniqueName: \"kubernetes.io/projected/abb35aa8-d003-42b1-8c94-a17afa96d114-kube-api-access-fw6zw\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:57:11.312395 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:11.305798 2580 generic.go:356] "Generic (PLEG): container finished" podID="15faee94-8483-4e7e-89df-543bb13771af" containerID="c583262e3f1bbe5ec0b10b7a0631ccf8d6fe678ba2af4fc720e12a343434b6a0" exitCode=0 Jul 21 21:57:11.312395 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:11.305891 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gg6fv" event={"ID":"15faee94-8483-4e7e-89df-543bb13771af","Type":"ContainerDied","Data":"c583262e3f1bbe5ec0b10b7a0631ccf8d6fe678ba2af4fc720e12a343434b6a0"} Jul 21 21:57:11.312395 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:11.305925 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gg6fv" event={"ID":"15faee94-8483-4e7e-89df-543bb13771af","Type":"ContainerDied","Data":"b92afe957f7c0b87edb60523ee9a88369d1da26c4b53ff22909d7e859922771a"} Jul 21 21:57:11.312395 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:11.305945 2580 scope.go:117] "RemoveContainer" containerID="c583262e3f1bbe5ec0b10b7a0631ccf8d6fe678ba2af4fc720e12a343434b6a0" Jul 21 21:57:11.312395 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:11.306106 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gg6fv" Jul 21 21:57:11.321328 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:11.320297 2580 generic.go:356] "Generic (PLEG): container finished" podID="abb35aa8-d003-42b1-8c94-a17afa96d114" containerID="868670ae904105e9c72ad73e604f3d0b8c4a7b3112d488a91dd3d272382999e4" exitCode=0 Jul 21 21:57:11.321328 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:11.320951 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pnqt6" Jul 21 21:57:11.321769 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:11.321733 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pnqt6" event={"ID":"abb35aa8-d003-42b1-8c94-a17afa96d114","Type":"ContainerDied","Data":"868670ae904105e9c72ad73e604f3d0b8c4a7b3112d488a91dd3d272382999e4"} Jul 21 21:57:11.321912 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:11.321786 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pnqt6" event={"ID":"abb35aa8-d003-42b1-8c94-a17afa96d114","Type":"ContainerDied","Data":"07090cb8ce84147364574e907ef6a2661c5dfed451d79382403027db4ff59163"} Jul 21 21:57:11.328845 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:11.328621 2580 scope.go:117] "RemoveContainer" containerID="c583262e3f1bbe5ec0b10b7a0631ccf8d6fe678ba2af4fc720e12a343434b6a0" Jul 21 21:57:11.328993 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:57:11.328975 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c583262e3f1bbe5ec0b10b7a0631ccf8d6fe678ba2af4fc720e12a343434b6a0\": container with ID starting with c583262e3f1bbe5ec0b10b7a0631ccf8d6fe678ba2af4fc720e12a343434b6a0 not found: ID does not exist" containerID="c583262e3f1bbe5ec0b10b7a0631ccf8d6fe678ba2af4fc720e12a343434b6a0" Jul 21 21:57:11.329044 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:11.329003 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c583262e3f1bbe5ec0b10b7a0631ccf8d6fe678ba2af4fc720e12a343434b6a0"} err="failed to get container status \"c583262e3f1bbe5ec0b10b7a0631ccf8d6fe678ba2af4fc720e12a343434b6a0\": rpc error: code = NotFound desc = could not find container \"c583262e3f1bbe5ec0b10b7a0631ccf8d6fe678ba2af4fc720e12a343434b6a0\": container with ID starting with c583262e3f1bbe5ec0b10b7a0631ccf8d6fe678ba2af4fc720e12a343434b6a0 not found: ID does not exist" Jul 21 21:57:11.329044 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:11.329020 2580 scope.go:117] "RemoveContainer" containerID="868670ae904105e9c72ad73e604f3d0b8c4a7b3112d488a91dd3d272382999e4" Jul 21 21:57:11.330105 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:11.330085 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gg6fv"] Jul 21 21:57:11.334023 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:11.333972 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gg6fv"] Jul 21 21:57:11.337219 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:11.337207 2580 scope.go:117] "RemoveContainer" containerID="868670ae904105e9c72ad73e604f3d0b8c4a7b3112d488a91dd3d272382999e4" Jul 21 21:57:11.337510 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:57:11.337490 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868670ae904105e9c72ad73e604f3d0b8c4a7b3112d488a91dd3d272382999e4\": container with ID starting with 868670ae904105e9c72ad73e604f3d0b8c4a7b3112d488a91dd3d272382999e4 not found: ID does not exist" containerID="868670ae904105e9c72ad73e604f3d0b8c4a7b3112d488a91dd3d272382999e4" Jul 21 21:57:11.337568 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:11.337515 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868670ae904105e9c72ad73e604f3d0b8c4a7b3112d488a91dd3d272382999e4"} err="failed to get container status \"868670ae904105e9c72ad73e604f3d0b8c4a7b3112d488a91dd3d272382999e4\": rpc error: code = NotFound desc = could not find container \"868670ae904105e9c72ad73e604f3d0b8c4a7b3112d488a91dd3d272382999e4\": container with ID starting with 868670ae904105e9c72ad73e604f3d0b8c4a7b3112d488a91dd3d272382999e4 not found: ID does not exist" Jul 21 21:57:11.344881 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:11.344860 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pnqt6"] Jul 21 21:57:11.350355 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:11.350337 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pnqt6"] Jul 21 21:57:12.779642 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:12.779614 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15faee94-8483-4e7e-89df-543bb13771af" path="/var/lib/kubelet/pods/15faee94-8483-4e7e-89df-543bb13771af/volumes" Jul 21 21:57:12.780104 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:12.780085 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abb35aa8-d003-42b1-8c94-a17afa96d114" path="/var/lib/kubelet/pods/abb35aa8-d003-42b1-8c94-a17afa96d114/volumes" Jul 21 21:57:15.100331 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.100297 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-696b77ff4c-4bj54"] Jul 21 21:57:15.101770 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.101743 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abb35aa8-d003-42b1-8c94-a17afa96d114" containerName="authorino" Jul 21 21:57:15.101931 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.101920 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="abb35aa8-d003-42b1-8c94-a17afa96d114" containerName="authorino" Jul 21 21:57:15.102097 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.102086 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15faee94-8483-4e7e-89df-543bb13771af" containerName="authorino" Jul 21 21:57:15.102173 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.102164 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15faee94-8483-4e7e-89df-543bb13771af" containerName="authorino" Jul 21 21:57:15.102809 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.102789 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="abb35aa8-d003-42b1-8c94-a17afa96d114" containerName="authorino" Jul 21 21:57:15.102946 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.102936 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15faee94-8483-4e7e-89df-543bb13771af" containerName="authorino" Jul 21 21:57:15.310791 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.310761 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-696b77ff4c-4bj54"] Jul 21 21:57:15.310791 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.310796 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55bc4db677-htrvj"] Jul 21 21:57:15.311025 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.310894 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-696b77ff4c-4bj54" Jul 21 21:57:15.314816 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.314794 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 21:57:15.314953 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.314800 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-f44sr\"" Jul 21 21:57:15.314953 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.314822 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 21:57:15.331558 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.331533 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55bc4db677-htrvj"] Jul 21 21:57:15.331657 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.331615 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55bc4db677-htrvj" Jul 21 21:57:15.428190 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.428122 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wzrt\" (UniqueName: \"kubernetes.io/projected/deed685d-7022-4a4e-8fa2-cc38091952ee-kube-api-access-7wzrt\") pod \"maas-controller-55bc4db677-htrvj\" (UID: \"deed685d-7022-4a4e-8fa2-cc38091952ee\") " pod="opendatahub/maas-controller-55bc4db677-htrvj" Jul 21 21:57:15.428190 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.428164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/deed685d-7022-4a4e-8fa2-cc38091952ee-webhook-cert\") pod \"maas-controller-55bc4db677-htrvj\" (UID: \"deed685d-7022-4a4e-8fa2-cc38091952ee\") " pod="opendatahub/maas-controller-55bc4db677-htrvj" Jul 21 21:57:15.428337 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.428195 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x987h\" (UniqueName: \"kubernetes.io/projected/86275017-79fb-41b6-9030-1faff08b21b8-kube-api-access-x987h\") pod \"maas-controller-696b77ff4c-4bj54\" (UID: \"86275017-79fb-41b6-9030-1faff08b21b8\") " pod="opendatahub/maas-controller-696b77ff4c-4bj54" Jul 21 21:57:15.428337 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.428304 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86275017-79fb-41b6-9030-1faff08b21b8-webhook-cert\") pod \"maas-controller-696b77ff4c-4bj54\" (UID: \"86275017-79fb-41b6-9030-1faff08b21b8\") " pod="opendatahub/maas-controller-696b77ff4c-4bj54" Jul 21 21:57:15.529205 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.529175 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/deed685d-7022-4a4e-8fa2-cc38091952ee-webhook-cert\") pod \"maas-controller-55bc4db677-htrvj\" (UID: \"deed685d-7022-4a4e-8fa2-cc38091952ee\") " pod="opendatahub/maas-controller-55bc4db677-htrvj" Jul 21 21:57:15.529355 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.529218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x987h\" (UniqueName: \"kubernetes.io/projected/86275017-79fb-41b6-9030-1faff08b21b8-kube-api-access-x987h\") pod \"maas-controller-696b77ff4c-4bj54\" (UID: \"86275017-79fb-41b6-9030-1faff08b21b8\") " pod="opendatahub/maas-controller-696b77ff4c-4bj54" Jul 21 21:57:15.529355 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.529269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86275017-79fb-41b6-9030-1faff08b21b8-webhook-cert\") pod \"maas-controller-696b77ff4c-4bj54\" (UID: \"86275017-79fb-41b6-9030-1faff08b21b8\") " pod="opendatahub/maas-controller-696b77ff4c-4bj54" Jul 21 21:57:15.529355 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.529304 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wzrt\" (UniqueName: \"kubernetes.io/projected/deed685d-7022-4a4e-8fa2-cc38091952ee-kube-api-access-7wzrt\") pod \"maas-controller-55bc4db677-htrvj\" (UID: \"deed685d-7022-4a4e-8fa2-cc38091952ee\") " pod="opendatahub/maas-controller-55bc4db677-htrvj" Jul 21 21:57:15.531780 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.531755 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86275017-79fb-41b6-9030-1faff08b21b8-webhook-cert\") pod \"maas-controller-696b77ff4c-4bj54\" (UID: \"86275017-79fb-41b6-9030-1faff08b21b8\") " pod="opendatahub/maas-controller-696b77ff4c-4bj54" Jul 21 21:57:15.531880 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.531813 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/deed685d-7022-4a4e-8fa2-cc38091952ee-webhook-cert\") pod \"maas-controller-55bc4db677-htrvj\" (UID: \"deed685d-7022-4a4e-8fa2-cc38091952ee\") " pod="opendatahub/maas-controller-55bc4db677-htrvj" Jul 21 21:57:15.538825 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.538801 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wzrt\" (UniqueName: \"kubernetes.io/projected/deed685d-7022-4a4e-8fa2-cc38091952ee-kube-api-access-7wzrt\") pod \"maas-controller-55bc4db677-htrvj\" (UID: \"deed685d-7022-4a4e-8fa2-cc38091952ee\") " pod="opendatahub/maas-controller-55bc4db677-htrvj" Jul 21 21:57:15.538825 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.538814 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x987h\" (UniqueName: \"kubernetes.io/projected/86275017-79fb-41b6-9030-1faff08b21b8-kube-api-access-x987h\") pod \"maas-controller-696b77ff4c-4bj54\" (UID: \"86275017-79fb-41b6-9030-1faff08b21b8\") " pod="opendatahub/maas-controller-696b77ff4c-4bj54" Jul 21 21:57:15.621998 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.621969 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-696b77ff4c-4bj54" Jul 21 21:57:15.641253 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.641220 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55bc4db677-htrvj" Jul 21 21:57:15.792675 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.792651 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-696b77ff4c-4bj54"] Jul 21 21:57:15.796139 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:57:15.796100 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86275017_79fb_41b6_9030_1faff08b21b8.slice/crio-c9ae77f1d7d3b4442665d04bb5f23867ab5f0c30439b187d4c6a60a390476be7 WatchSource:0}: Error finding container c9ae77f1d7d3b4442665d04bb5f23867ab5f0c30439b187d4c6a60a390476be7: Status 404 returned error can't find the container with id c9ae77f1d7d3b4442665d04bb5f23867ab5f0c30439b187d4c6a60a390476be7 Jul 21 21:57:15.819238 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:15.819217 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55bc4db677-htrvj"] Jul 21 21:57:15.821245 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:57:15.821227 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeed685d_7022_4a4e_8fa2_cc38091952ee.slice/crio-d3b412890b2610de12561aaa8fefadec517819b4980ecfd0c39b6434e5576629 WatchSource:0}: Error finding container d3b412890b2610de12561aaa8fefadec517819b4980ecfd0c39b6434e5576629: Status 404 returned error can't find the container with id d3b412890b2610de12561aaa8fefadec517819b4980ecfd0c39b6434e5576629 Jul 21 21:57:16.350099 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:16.350051 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-696b77ff4c-4bj54" event={"ID":"86275017-79fb-41b6-9030-1faff08b21b8","Type":"ContainerStarted","Data":"c9ae77f1d7d3b4442665d04bb5f23867ab5f0c30439b187d4c6a60a390476be7"} Jul 21 21:57:16.351845 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:16.351817 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55bc4db677-htrvj" event={"ID":"deed685d-7022-4a4e-8fa2-cc38091952ee","Type":"ContainerStarted","Data":"d3b412890b2610de12561aaa8fefadec517819b4980ecfd0c39b6434e5576629"} Jul 21 21:57:18.364486 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:18.364456 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55bc4db677-htrvj" event={"ID":"deed685d-7022-4a4e-8fa2-cc38091952ee","Type":"ContainerStarted","Data":"c8c5bb6e67c1b5c562e4b9bb1672ad6052ee49c320b979a8f3f32cd170d16a53"} Jul 21 21:57:18.364888 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:18.364544 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55bc4db677-htrvj" Jul 21 21:57:18.366180 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:18.366148 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-696b77ff4c-4bj54" event={"ID":"86275017-79fb-41b6-9030-1faff08b21b8","Type":"ContainerStarted","Data":"8233c6f8046d53b5ea852ffa4b0e6b4e3edf5e69efb93866359f3ec9832c9276"} Jul 21 21:57:18.366285 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:18.366278 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-696b77ff4c-4bj54" Jul 21 21:57:18.393889 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:18.393841 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55bc4db677-htrvj" podStartSLOduration=0.965064867 podStartE2EDuration="3.393828598s" totalImagesPullingTime="2.428763731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:57:18.391987512 +0000 UTC m=+516.155221775" watchObservedRunningTime="2026-07-21 21:57:18.393828598 +0000 UTC m=+516.157062902" Jul 21 21:57:18.416165 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:18.416121 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-696b77ff4c-4bj54" podStartSLOduration=0.958665334 podStartE2EDuration="3.416107906s" totalImagesPullingTime="2.457442572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:57:18.414407193 +0000 UTC m=+516.177641455" watchObservedRunningTime="2026-07-21 21:57:18.416107906 +0000 UTC m=+516.179342166" Jul 21 21:57:29.374095 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:29.374062 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55bc4db677-htrvj" Jul 21 21:57:29.374609 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:29.374232 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-696b77ff4c-4bj54" Jul 21 21:57:29.440269 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:29.440094 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-696b77ff4c-4bj54"] Jul 21 21:57:29.440470 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:29.440353 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-696b77ff4c-4bj54" podUID="86275017-79fb-41b6-9030-1faff08b21b8" containerName="manager" containerID="cri-o://8233c6f8046d53b5ea852ffa4b0e6b4e3edf5e69efb93866359f3ec9832c9276" gracePeriod=30 Jul 21 21:57:29.689474 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:29.689450 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-696b77ff4c-4bj54" Jul 21 21:57:29.863650 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:29.863615 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86275017-79fb-41b6-9030-1faff08b21b8-webhook-cert\") pod \"86275017-79fb-41b6-9030-1faff08b21b8\" (UID: \"86275017-79fb-41b6-9030-1faff08b21b8\") " Jul 21 21:57:29.863819 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:29.863750 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x987h\" (UniqueName: \"kubernetes.io/projected/86275017-79fb-41b6-9030-1faff08b21b8-kube-api-access-x987h\") pod \"86275017-79fb-41b6-9030-1faff08b21b8\" (UID: \"86275017-79fb-41b6-9030-1faff08b21b8\") " Jul 21 21:57:29.865978 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:29.865940 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86275017-79fb-41b6-9030-1faff08b21b8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "86275017-79fb-41b6-9030-1faff08b21b8" (UID: "86275017-79fb-41b6-9030-1faff08b21b8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:57:29.866091 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:29.866009 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86275017-79fb-41b6-9030-1faff08b21b8-kube-api-access-x987h" (OuterVolumeSpecName: "kube-api-access-x987h") pod "86275017-79fb-41b6-9030-1faff08b21b8" (UID: "86275017-79fb-41b6-9030-1faff08b21b8"). InnerVolumeSpecName "kube-api-access-x987h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:57:29.964993 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:29.964897 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x987h\" (UniqueName: \"kubernetes.io/projected/86275017-79fb-41b6-9030-1faff08b21b8-kube-api-access-x987h\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:57:29.964993 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:29.964927 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86275017-79fb-41b6-9030-1faff08b21b8-webhook-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 21:57:30.431025 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:30.430987 2580 generic.go:356] "Generic (PLEG): container finished" podID="86275017-79fb-41b6-9030-1faff08b21b8" containerID="8233c6f8046d53b5ea852ffa4b0e6b4e3edf5e69efb93866359f3ec9832c9276" exitCode=0 Jul 21 21:57:30.431025 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:30.431029 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-696b77ff4c-4bj54" Jul 21 21:57:30.431521 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:30.431076 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-696b77ff4c-4bj54" event={"ID":"86275017-79fb-41b6-9030-1faff08b21b8","Type":"ContainerDied","Data":"8233c6f8046d53b5ea852ffa4b0e6b4e3edf5e69efb93866359f3ec9832c9276"} Jul 21 21:57:30.431521 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:30.431120 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-696b77ff4c-4bj54" event={"ID":"86275017-79fb-41b6-9030-1faff08b21b8","Type":"ContainerDied","Data":"c9ae77f1d7d3b4442665d04bb5f23867ab5f0c30439b187d4c6a60a390476be7"} Jul 21 21:57:30.431521 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:30.431140 2580 scope.go:117] "RemoveContainer" containerID="8233c6f8046d53b5ea852ffa4b0e6b4e3edf5e69efb93866359f3ec9832c9276" Jul 21 21:57:30.440544 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:30.440529 2580 scope.go:117] "RemoveContainer" containerID="8233c6f8046d53b5ea852ffa4b0e6b4e3edf5e69efb93866359f3ec9832c9276" Jul 21 21:57:30.440797 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:57:30.440779 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8233c6f8046d53b5ea852ffa4b0e6b4e3edf5e69efb93866359f3ec9832c9276\": container with ID starting with 8233c6f8046d53b5ea852ffa4b0e6b4e3edf5e69efb93866359f3ec9832c9276 not found: ID does not exist" containerID="8233c6f8046d53b5ea852ffa4b0e6b4e3edf5e69efb93866359f3ec9832c9276" Jul 21 21:57:30.440843 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:30.440808 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8233c6f8046d53b5ea852ffa4b0e6b4e3edf5e69efb93866359f3ec9832c9276"} err="failed to get container status \"8233c6f8046d53b5ea852ffa4b0e6b4e3edf5e69efb93866359f3ec9832c9276\": rpc error: code = NotFound desc = could not find container \"8233c6f8046d53b5ea852ffa4b0e6b4e3edf5e69efb93866359f3ec9832c9276\": container with ID starting with 8233c6f8046d53b5ea852ffa4b0e6b4e3edf5e69efb93866359f3ec9832c9276 not found: ID does not exist" Jul 21 21:57:30.453089 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:30.453060 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-696b77ff4c-4bj54"] Jul 21 21:57:30.461281 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:30.461263 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-696b77ff4c-4bj54"] Jul 21 21:57:30.778960 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:30.778921 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86275017-79fb-41b6-9030-1faff08b21b8" path="/var/lib/kubelet/pods/86275017-79fb-41b6-9030-1faff08b21b8/volumes" Jul 21 21:57:51.381165 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.381136 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-h29c8"] Jul 21 21:57:51.381618 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.381541 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86275017-79fb-41b6-9030-1faff08b21b8" containerName="manager" Jul 21 21:57:51.381618 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.381552 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="86275017-79fb-41b6-9030-1faff08b21b8" containerName="manager" Jul 21 21:57:51.381719 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.381661 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="86275017-79fb-41b6-9030-1faff08b21b8" containerName="manager" Jul 21 21:57:51.384716 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.384698 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-h29c8" Jul 21 21:57:51.387297 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.387275 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-fcgv9\"" Jul 21 21:57:51.387424 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.387301 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 21 21:57:51.395466 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.395448 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-h29c8"] Jul 21 21:57:51.405647 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.405627 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-pv4cl"] Jul 21 21:57:51.408397 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.408363 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-pv4cl" Jul 21 21:57:51.418611 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.418590 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-pv4cl"] Jul 21 21:57:51.461974 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.461947 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b4ffb7-f0c9-4cc2-a090-ccbafe863a91-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-h29c8\" (UID: \"50b4ffb7-f0c9-4cc2-a090-ccbafe863a91\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-h29c8" Jul 21 21:57:51.462091 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.462023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nng5l\" (UniqueName: \"kubernetes.io/projected/ea66ae8c-0078-4574-80a9-0ae2dddec6ca-kube-api-access-nng5l\") pod \"payload-processing-5bf47cd988-pv4cl\" (UID: \"ea66ae8c-0078-4574-80a9-0ae2dddec6ca\") " pod="openshift-ingress/payload-processing-5bf47cd988-pv4cl" Jul 21 21:57:51.462091 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.462054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnfrw\" (UniqueName: \"kubernetes.io/projected/50b4ffb7-f0c9-4cc2-a090-ccbafe863a91-kube-api-access-rnfrw\") pod \"payload-pre-processing-7677b774bb-h29c8\" (UID: \"50b4ffb7-f0c9-4cc2-a090-ccbafe863a91\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-h29c8" Jul 21 21:57:51.462161 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.462094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea66ae8c-0078-4574-80a9-0ae2dddec6ca-plugins-config-volume\") pod \"payload-processing-5bf47cd988-pv4cl\" (UID: \"ea66ae8c-0078-4574-80a9-0ae2dddec6ca\") " pod="openshift-ingress/payload-processing-5bf47cd988-pv4cl" Jul 21 21:57:51.562670 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.562637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nng5l\" (UniqueName: \"kubernetes.io/projected/ea66ae8c-0078-4574-80a9-0ae2dddec6ca-kube-api-access-nng5l\") pod \"payload-processing-5bf47cd988-pv4cl\" (UID: \"ea66ae8c-0078-4574-80a9-0ae2dddec6ca\") " pod="openshift-ingress/payload-processing-5bf47cd988-pv4cl" Jul 21 21:57:51.562840 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.562697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnfrw\" (UniqueName: \"kubernetes.io/projected/50b4ffb7-f0c9-4cc2-a090-ccbafe863a91-kube-api-access-rnfrw\") pod \"payload-pre-processing-7677b774bb-h29c8\" (UID: \"50b4ffb7-f0c9-4cc2-a090-ccbafe863a91\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-h29c8" Jul 21 21:57:51.562840 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.562758 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea66ae8c-0078-4574-80a9-0ae2dddec6ca-plugins-config-volume\") pod \"payload-processing-5bf47cd988-pv4cl\" (UID: \"ea66ae8c-0078-4574-80a9-0ae2dddec6ca\") " pod="openshift-ingress/payload-processing-5bf47cd988-pv4cl" Jul 21 21:57:51.562840 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.562812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b4ffb7-f0c9-4cc2-a090-ccbafe863a91-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-h29c8\" (UID: \"50b4ffb7-f0c9-4cc2-a090-ccbafe863a91\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-h29c8" Jul 21 21:57:51.563671 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.563649 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea66ae8c-0078-4574-80a9-0ae2dddec6ca-plugins-config-volume\") pod \"payload-processing-5bf47cd988-pv4cl\" (UID: \"ea66ae8c-0078-4574-80a9-0ae2dddec6ca\") " pod="openshift-ingress/payload-processing-5bf47cd988-pv4cl" Jul 21 21:57:51.563778 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.563756 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b4ffb7-f0c9-4cc2-a090-ccbafe863a91-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-h29c8\" (UID: \"50b4ffb7-f0c9-4cc2-a090-ccbafe863a91\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-h29c8" Jul 21 21:57:51.571796 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.571767 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnfrw\" (UniqueName: \"kubernetes.io/projected/50b4ffb7-f0c9-4cc2-a090-ccbafe863a91-kube-api-access-rnfrw\") pod \"payload-pre-processing-7677b774bb-h29c8\" (UID: \"50b4ffb7-f0c9-4cc2-a090-ccbafe863a91\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-h29c8" Jul 21 21:57:51.571921 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.571869 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nng5l\" (UniqueName: \"kubernetes.io/projected/ea66ae8c-0078-4574-80a9-0ae2dddec6ca-kube-api-access-nng5l\") pod \"payload-processing-5bf47cd988-pv4cl\" (UID: \"ea66ae8c-0078-4574-80a9-0ae2dddec6ca\") " pod="openshift-ingress/payload-processing-5bf47cd988-pv4cl" Jul 21 21:57:51.696092 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.696008 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-h29c8" Jul 21 21:57:51.719316 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.719064 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-pv4cl" Jul 21 21:57:51.852234 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.852207 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-h29c8"] Jul 21 21:57:51.853413 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:57:51.853357 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b4ffb7_f0c9_4cc2_a090_ccbafe863a91.slice/crio-e85bc599127321a27df2514361a28a27a2e00f8a99477cc8eaa054afe166e5d6 WatchSource:0}: Error finding container e85bc599127321a27df2514361a28a27a2e00f8a99477cc8eaa054afe166e5d6: Status 404 returned error can't find the container with id e85bc599127321a27df2514361a28a27a2e00f8a99477cc8eaa054afe166e5d6 Jul 21 21:57:51.875663 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:51.875643 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-pv4cl"] Jul 21 21:57:51.877424 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:57:51.877404 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea66ae8c_0078_4574_80a9_0ae2dddec6ca.slice/crio-524acc6671ad4adbc7c6883870efcf8c2a5184a57c7c360e66952c3c4f51b714 WatchSource:0}: Error finding container 524acc6671ad4adbc7c6883870efcf8c2a5184a57c7c360e66952c3c4f51b714: Status 404 returned error can't find the container with id 524acc6671ad4adbc7c6883870efcf8c2a5184a57c7c360e66952c3c4f51b714 Jul 21 21:57:52.558307 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:52.558223 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-h29c8" event={"ID":"50b4ffb7-f0c9-4cc2-a090-ccbafe863a91","Type":"ContainerStarted","Data":"e85bc599127321a27df2514361a28a27a2e00f8a99477cc8eaa054afe166e5d6"} Jul 21 21:57:52.560451 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:52.560421 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-pv4cl" event={"ID":"ea66ae8c-0078-4574-80a9-0ae2dddec6ca","Type":"ContainerStarted","Data":"524acc6671ad4adbc7c6883870efcf8c2a5184a57c7c360e66952c3c4f51b714"} Jul 21 21:57:54.577694 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:54.577653 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-h29c8" event={"ID":"50b4ffb7-f0c9-4cc2-a090-ccbafe863a91","Type":"ContainerStarted","Data":"beddb943dd710db1b08175f4c80ce564d220694077cce3725815bb2edf2ae639"} Jul 21 21:57:54.578138 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:54.577785 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-h29c8" Jul 21 21:57:54.578843 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:54.578810 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-pv4cl" event={"ID":"ea66ae8c-0078-4574-80a9-0ae2dddec6ca","Type":"ContainerStarted","Data":"bc1abfea82ecc18d4a9bad8b663af1b33abda3789dbf9ed84f442112a5e1f188"} Jul 21 21:57:54.578978 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:54.578961 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-pv4cl" Jul 21 21:57:54.596313 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:54.596258 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-h29c8" podStartSLOduration=1.401364766 podStartE2EDuration="3.59624047s" totalImagesPullingTime="2.194875704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:57:54.594549999 +0000 UTC m=+552.357784261" watchObservedRunningTime="2026-07-21 21:57:54.59624047 +0000 UTC m=+552.359474735" Jul 21 21:57:54.611047 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:54.611004 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-pv4cl" podStartSLOduration=1.428790728 podStartE2EDuration="3.610991766s" totalImagesPullingTime="2.182201038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:57:54.609530178 +0000 UTC m=+552.372764441" watchObservedRunningTime="2026-07-21 21:57:54.610991766 +0000 UTC m=+552.374226028" Jul 21 21:57:59.607969 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:59.607933 2580 generic.go:356] "Generic (PLEG): container finished" podID="de321670-e3b1-4976-9799-1bcb1e8d4823" containerID="430e259afad8c96efc9756eed2202f3b673fbbaf6b6e0eb74b09e88c93b40b7a" exitCode=0 Jul 21 21:57:59.608579 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:59.608004 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" event={"ID":"de321670-e3b1-4976-9799-1bcb1e8d4823","Type":"ContainerDied","Data":"430e259afad8c96efc9756eed2202f3b673fbbaf6b6e0eb74b09e88c93b40b7a"} Jul 21 21:57:59.608579 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:59.608447 2580 scope.go:117] "RemoveContainer" containerID="430e259afad8c96efc9756eed2202f3b673fbbaf6b6e0eb74b09e88c93b40b7a" Jul 21 21:57:59.609516 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:59.609486 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 21:57:59.609592 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:59.609572 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 21:57:59.609632 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:57:59.609609 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 21:58:00.614589 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:00.614557 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" event={"ID":"de321670-e3b1-4976-9799-1bcb1e8d4823","Type":"ContainerStarted","Data":"76b9c83caf78f4c98d72bc654f19d5b8f71195a2881b2b73286c1843087f26da"} Jul 21 21:58:00.694726 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:00.694687 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:58:00.695027 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:00.695002 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bwxc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 21 21:58:00.695088 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:00.695063 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" podUID="de321670-e3b1-4976-9799-1bcb1e8d4823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 21 21:58:01.695481 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:01.695437 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bwxc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 21 21:58:01.695887 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:01.695526 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" podUID="de321670-e3b1-4976-9799-1bcb1e8d4823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 21 21:58:02.695348 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:02.695302 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bwxc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 21 21:58:02.695550 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:02.695411 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" podUID="de321670-e3b1-4976-9799-1bcb1e8d4823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 21 21:58:03.695775 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:03.695737 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bwxc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 21 21:58:03.696184 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:03.695798 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" podUID="de321670-e3b1-4976-9799-1bcb1e8d4823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 21 21:58:04.699315 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:04.699285 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:58:04.699728 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:04.699693 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:58:04.700464 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:04.700448 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bwxc6" Jul 21 21:58:05.590143 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:05.590113 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-pv4cl" Jul 21 21:58:06.595099 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:06.595068 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-h29c8" Jul 21 21:58:14.437449 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.437418 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r"] Jul 21 21:58:14.440107 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.440090 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:14.447260 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:58:14.447232 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-distinct-2-simulated-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" type="*v1.Secret" Jul 21 21:58:14.448211 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:58:14.448188 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" podUID="800d232d-d480-472a-8d5b-4f2d2290007d" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:14.449170 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:58:14.449145 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-kccl4\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"llm\"/\"default-dockercfg-kccl4\"" type="*v1.Secret" Jul 21 21:58:14.449225 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:58:14.449173 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 21 21:58:14.449225 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:58:14.449212 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-158.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-135-158.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 21 21:58:14.478302 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.478265 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r"] Jul 21 21:58:14.517391 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.517349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/800d232d-d480-472a-8d5b-4f2d2290007d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:14.517549 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.517477 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/800d232d-d480-472a-8d5b-4f2d2290007d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:14.517549 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.517507 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/800d232d-d480-472a-8d5b-4f2d2290007d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:14.517628 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.517575 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/800d232d-d480-472a-8d5b-4f2d2290007d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:14.517628 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.517617 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/800d232d-d480-472a-8d5b-4f2d2290007d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:14.517690 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.517636 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9849w\" (UniqueName: \"kubernetes.io/projected/800d232d-d480-472a-8d5b-4f2d2290007d-kube-api-access-9849w\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:14.618136 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.618100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/800d232d-d480-472a-8d5b-4f2d2290007d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:14.618350 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.618165 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/800d232d-d480-472a-8d5b-4f2d2290007d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:14.618350 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.618181 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/800d232d-d480-472a-8d5b-4f2d2290007d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:14.618350 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.618221 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/800d232d-d480-472a-8d5b-4f2d2290007d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:14.618350 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.618258 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/800d232d-d480-472a-8d5b-4f2d2290007d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:14.618350 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.618282 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9849w\" (UniqueName: \"kubernetes.io/projected/800d232d-d480-472a-8d5b-4f2d2290007d-kube-api-access-9849w\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:14.618685 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.618662 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/800d232d-d480-472a-8d5b-4f2d2290007d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:14.618762 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.618741 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/800d232d-d480-472a-8d5b-4f2d2290007d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:14.618806 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.618784 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/800d232d-d480-472a-8d5b-4f2d2290007d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:14.620817 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:14.620794 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/800d232d-d480-472a-8d5b-4f2d2290007d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:15.303475 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:15.303442 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 21:58:15.590577 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:15.590496 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kccl4\"" Jul 21 21:58:15.619326 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:58:15.619290 2580 secret.go:189] Couldn't get secret llm/e2e-distinct-2-simulated-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition Jul 21 21:58:15.619530 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:58:15.619429 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/800d232d-d480-472a-8d5b-4f2d2290007d-tls-certs podName:800d232d-d480-472a-8d5b-4f2d2290007d nodeName:}" failed. No retries permitted until 2026-07-21 21:58:16.119406791 +0000 UTC m=+573.882641034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/800d232d-d480-472a-8d5b-4f2d2290007d-tls-certs") pod "e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" (UID: "800d232d-d480-472a-8d5b-4f2d2290007d") : failed to sync secret cache: timed out waiting for the condition Jul 21 21:58:15.629157 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:58:15.629134 2580 projected.go:291] Couldn't get configMap llm/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Jul 21 21:58:15.629272 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:58:15.629175 2580 projected.go:196] Error preparing data for projected volume kube-api-access-9849w for pod llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r: failed to sync configmap cache: timed out waiting for the condition Jul 21 21:58:15.629272 ip-10-0-135-158 kubenswrapper[2580]: E0721 21:58:15.629241 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/800d232d-d480-472a-8d5b-4f2d2290007d-kube-api-access-9849w podName:800d232d-d480-472a-8d5b-4f2d2290007d nodeName:}" failed. No retries permitted until 2026-07-21 21:58:16.129220311 +0000 UTC m=+573.892454550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9849w" (UniqueName: "kubernetes.io/projected/800d232d-d480-472a-8d5b-4f2d2290007d-kube-api-access-9849w") pod "e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" (UID: "800d232d-d480-472a-8d5b-4f2d2290007d") : failed to sync configmap cache: timed out waiting for the condition Jul 21 21:58:15.661717 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:15.661680 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 21:58:15.823592 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:15.823559 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 21 21:58:16.139564 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:16.139525 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/800d232d-d480-472a-8d5b-4f2d2290007d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:16.139765 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:16.139619 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9849w\" (UniqueName: \"kubernetes.io/projected/800d232d-d480-472a-8d5b-4f2d2290007d-kube-api-access-9849w\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:16.142214 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:16.142179 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/800d232d-d480-472a-8d5b-4f2d2290007d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:16.142328 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:16.142281 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9849w\" (UniqueName: \"kubernetes.io/projected/800d232d-d480-472a-8d5b-4f2d2290007d-kube-api-access-9849w\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r\" (UID: \"800d232d-d480-472a-8d5b-4f2d2290007d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:16.250089 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:16.250054 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:16.406208 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:16.406174 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r"] Jul 21 21:58:16.407041 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:58:16.407013 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800d232d_d480_472a_8d5b_4f2d2290007d.slice/crio-1cdee3aba6a8f871ba3882f0aeef8a57b87b2da446ac9b9d3e0102f9401fe761 WatchSource:0}: Error finding container 1cdee3aba6a8f871ba3882f0aeef8a57b87b2da446ac9b9d3e0102f9401fe761: Status 404 returned error can't find the container with id 1cdee3aba6a8f871ba3882f0aeef8a57b87b2da446ac9b9d3e0102f9401fe761 Jul 21 21:58:16.702708 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:16.702544 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" event={"ID":"800d232d-d480-472a-8d5b-4f2d2290007d","Type":"ContainerStarted","Data":"1cdee3aba6a8f871ba3882f0aeef8a57b87b2da446ac9b9d3e0102f9401fe761"} Jul 21 21:58:17.710635 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:17.710567 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" event={"ID":"800d232d-d480-472a-8d5b-4f2d2290007d","Type":"ContainerStarted","Data":"0c1071d8e4662012612ed7661f535c6e691be0e8a17844a407ed5dd496968943"} Jul 21 21:58:17.734047 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:17.733989 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" podStartSLOduration=2.609016408 podStartE2EDuration="3.733972555s" totalImagesPullingTime="1.124956147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:58:17.732365232 +0000 UTC m=+575.495599493" watchObservedRunningTime="2026-07-21 21:58:17.733972555 +0000 UTC m=+575.497206815" Jul 21 21:58:26.250791 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:26.250750 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:26.251257 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:26.250822 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:26.263604 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:26.263578 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:58:26.770852 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:26.770820 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r" Jul 21 21:59:00.000022 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:58:59.999968 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b"] Jul 21 21:59:00.003820 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.003799 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.006592 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.006573 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 21 21:59:00.016307 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.016279 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b"] Jul 21 21:59:00.074651 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.074618 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v24lq\" (UniqueName: \"kubernetes.io/projected/d3918073-933b-4335-9171-1c070291e21c-kube-api-access-v24lq\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5rc4b\" (UID: \"d3918073-933b-4335-9171-1c070291e21c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.074841 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.074662 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3918073-933b-4335-9171-1c070291e21c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5rc4b\" (UID: \"d3918073-933b-4335-9171-1c070291e21c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.074841 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.074747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3918073-933b-4335-9171-1c070291e21c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5rc4b\" (UID: \"d3918073-933b-4335-9171-1c070291e21c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.074975 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.074878 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3918073-933b-4335-9171-1c070291e21c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5rc4b\" (UID: \"d3918073-933b-4335-9171-1c070291e21c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.074975 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.074953 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3918073-933b-4335-9171-1c070291e21c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5rc4b\" (UID: \"d3918073-933b-4335-9171-1c070291e21c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.075063 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.074984 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3918073-933b-4335-9171-1c070291e21c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5rc4b\" (UID: \"d3918073-933b-4335-9171-1c070291e21c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.175754 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.175721 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3918073-933b-4335-9171-1c070291e21c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5rc4b\" (UID: \"d3918073-933b-4335-9171-1c070291e21c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.175918 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.175781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3918073-933b-4335-9171-1c070291e21c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5rc4b\" (UID: \"d3918073-933b-4335-9171-1c070291e21c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.175918 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.175807 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3918073-933b-4335-9171-1c070291e21c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5rc4b\" (UID: \"d3918073-933b-4335-9171-1c070291e21c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.175918 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.175851 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v24lq\" (UniqueName: \"kubernetes.io/projected/d3918073-933b-4335-9171-1c070291e21c-kube-api-access-v24lq\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5rc4b\" (UID: \"d3918073-933b-4335-9171-1c070291e21c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.175918 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.175882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3918073-933b-4335-9171-1c070291e21c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5rc4b\" (UID: \"d3918073-933b-4335-9171-1c070291e21c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.176125 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.175924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3918073-933b-4335-9171-1c070291e21c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5rc4b\" (UID: \"d3918073-933b-4335-9171-1c070291e21c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.176168 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.176144 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3918073-933b-4335-9171-1c070291e21c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5rc4b\" (UID: \"d3918073-933b-4335-9171-1c070291e21c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.176213 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.176196 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3918073-933b-4335-9171-1c070291e21c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5rc4b\" (UID: \"d3918073-933b-4335-9171-1c070291e21c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.176286 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.176271 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3918073-933b-4335-9171-1c070291e21c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5rc4b\" (UID: \"d3918073-933b-4335-9171-1c070291e21c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.178154 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.178119 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3918073-933b-4335-9171-1c070291e21c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5rc4b\" (UID: \"d3918073-933b-4335-9171-1c070291e21c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.178366 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.178348 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3918073-933b-4335-9171-1c070291e21c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5rc4b\" (UID: \"d3918073-933b-4335-9171-1c070291e21c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.185054 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.185033 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v24lq\" (UniqueName: \"kubernetes.io/projected/d3918073-933b-4335-9171-1c070291e21c-kube-api-access-v24lq\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5rc4b\" (UID: \"d3918073-933b-4335-9171-1c070291e21c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.314543 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.314458 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:00.452862 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.452835 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b"] Jul 21 21:59:00.454727 ip-10-0-135-158 kubenswrapper[2580]: W0721 21:59:00.454701 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3918073_933b_4335_9171_1c070291e21c.slice/crio-3ea29d14a37ff2d88b75b0340bfcbfc718921333cb78c5a7d90e6371a5b74d1e WatchSource:0}: Error finding container 3ea29d14a37ff2d88b75b0340bfcbfc718921333cb78c5a7d90e6371a5b74d1e: Status 404 returned error can't find the container with id 3ea29d14a37ff2d88b75b0340bfcbfc718921333cb78c5a7d90e6371a5b74d1e Jul 21 21:59:00.456245 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.456227 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:59:00.931415 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.931306 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" event={"ID":"d3918073-933b-4335-9171-1c070291e21c","Type":"ContainerStarted","Data":"c86ffc644093cf9fbe9101b34d9b72d1409795926bcd2d4832cdc1cf5999865e"} Jul 21 21:59:00.931415 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.931354 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" event={"ID":"d3918073-933b-4335-9171-1c070291e21c","Type":"ContainerStarted","Data":"3ea29d14a37ff2d88b75b0340bfcbfc718921333cb78c5a7d90e6371a5b74d1e"} Jul 21 21:59:00.954567 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:00.954524 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" podStartSLOduration=1.777933615 podStartE2EDuration="1.954509203s" totalImagesPullingTime="176.575588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:59:00.951944322 +0000 UTC m=+618.715178583" watchObservedRunningTime="2026-07-21 21:59:00.954509203 +0000 UTC m=+618.717743463" Jul 21 21:59:10.314581 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:10.314544 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:10.315036 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:10.314644 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:10.327454 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:10.327422 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 21:59:10.997426 ip-10-0-135-158 kubenswrapper[2580]: I0721 21:59:10.997393 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5rc4b" Jul 21 22:00:47.134760 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:47.134681 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5569887bcc-swrx8"] Jul 21 22:00:47.135261 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:47.134864 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5569887bcc-swrx8" podUID="cc169b0d-34ca-4a21-84c3-16fe6de2174a" containerName="authorino" containerID="cri-o://2197161167d359214197b909ae02f0372348bcf31aa5a49c0fd915b1b889aba8" gracePeriod=30 Jul 21 22:00:47.385992 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:47.385934 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5569887bcc-swrx8" Jul 21 22:00:47.491088 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:47.491055 2580 generic.go:356] "Generic (PLEG): container finished" podID="cc169b0d-34ca-4a21-84c3-16fe6de2174a" containerID="2197161167d359214197b909ae02f0372348bcf31aa5a49c0fd915b1b889aba8" exitCode=0 Jul 21 22:00:47.491290 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:47.491105 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5569887bcc-swrx8" Jul 21 22:00:47.491290 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:47.491151 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5569887bcc-swrx8" event={"ID":"cc169b0d-34ca-4a21-84c3-16fe6de2174a","Type":"ContainerDied","Data":"2197161167d359214197b909ae02f0372348bcf31aa5a49c0fd915b1b889aba8"} Jul 21 22:00:47.491290 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:47.491198 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5569887bcc-swrx8" event={"ID":"cc169b0d-34ca-4a21-84c3-16fe6de2174a","Type":"ContainerDied","Data":"1922fe9d67215d76613b21bc015a8e27e1735ecddb7c77d6a61209d015ef77a2"} Jul 21 22:00:47.491290 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:47.491220 2580 scope.go:117] "RemoveContainer" containerID="2197161167d359214197b909ae02f0372348bcf31aa5a49c0fd915b1b889aba8" Jul 21 22:00:47.500284 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:47.500266 2580 scope.go:117] "RemoveContainer" containerID="2197161167d359214197b909ae02f0372348bcf31aa5a49c0fd915b1b889aba8" Jul 21 22:00:47.500581 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:00:47.500561 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2197161167d359214197b909ae02f0372348bcf31aa5a49c0fd915b1b889aba8\": container with ID starting with 2197161167d359214197b909ae02f0372348bcf31aa5a49c0fd915b1b889aba8 not found: ID does not exist" containerID="2197161167d359214197b909ae02f0372348bcf31aa5a49c0fd915b1b889aba8" Jul 21 22:00:47.500642 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:47.500590 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2197161167d359214197b909ae02f0372348bcf31aa5a49c0fd915b1b889aba8"} err="failed to get container status \"2197161167d359214197b909ae02f0372348bcf31aa5a49c0fd915b1b889aba8\": rpc error: code = NotFound desc = could not find container \"2197161167d359214197b909ae02f0372348bcf31aa5a49c0fd915b1b889aba8\": container with ID starting with 2197161167d359214197b909ae02f0372348bcf31aa5a49c0fd915b1b889aba8 not found: ID does not exist" Jul 21 22:00:47.516193 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:47.516166 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc169b0d-34ca-4a21-84c3-16fe6de2174a-tls-cert\") pod \"cc169b0d-34ca-4a21-84c3-16fe6de2174a\" (UID: \"cc169b0d-34ca-4a21-84c3-16fe6de2174a\") " Jul 21 22:00:47.516314 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:47.516245 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74czh\" (UniqueName: \"kubernetes.io/projected/cc169b0d-34ca-4a21-84c3-16fe6de2174a-kube-api-access-74czh\") pod \"cc169b0d-34ca-4a21-84c3-16fe6de2174a\" (UID: \"cc169b0d-34ca-4a21-84c3-16fe6de2174a\") " Jul 21 22:00:47.518367 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:47.518334 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc169b0d-34ca-4a21-84c3-16fe6de2174a-kube-api-access-74czh" (OuterVolumeSpecName: "kube-api-access-74czh") pod "cc169b0d-34ca-4a21-84c3-16fe6de2174a" (UID: "cc169b0d-34ca-4a21-84c3-16fe6de2174a"). InnerVolumeSpecName "kube-api-access-74czh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:00:47.528000 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:47.527970 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc169b0d-34ca-4a21-84c3-16fe6de2174a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "cc169b0d-34ca-4a21-84c3-16fe6de2174a" (UID: "cc169b0d-34ca-4a21-84c3-16fe6de2174a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:00:47.617566 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:47.617524 2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc169b0d-34ca-4a21-84c3-16fe6de2174a-tls-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:00:47.617566 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:47.617566 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74czh\" (UniqueName: \"kubernetes.io/projected/cc169b0d-34ca-4a21-84c3-16fe6de2174a-kube-api-access-74czh\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:00:47.823663 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:47.823629 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5569887bcc-swrx8"] Jul 21 22:00:47.833754 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:47.833729 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5569887bcc-swrx8"] Jul 21 22:00:48.780522 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:00:48.780490 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc169b0d-34ca-4a21-84c3-16fe6de2174a" path="/var/lib/kubelet/pods/cc169b0d-34ca-4a21-84c3-16fe6de2174a/volumes" Jul 21 22:01:35.222686 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:35.222651 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55bc4db677-htrvj"] Jul 21 22:01:35.223165 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:35.222852 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-55bc4db677-htrvj" podUID="deed685d-7022-4a4e-8fa2-cc38091952ee" containerName="manager" containerID="cri-o://c8c5bb6e67c1b5c562e4b9bb1672ad6052ee49c320b979a8f3f32cd170d16a53" gracePeriod=30 Jul 21 22:01:35.478894 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:35.478832 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55bc4db677-htrvj" Jul 21 22:01:35.588005 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:35.587976 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/deed685d-7022-4a4e-8fa2-cc38091952ee-webhook-cert\") pod \"deed685d-7022-4a4e-8fa2-cc38091952ee\" (UID: \"deed685d-7022-4a4e-8fa2-cc38091952ee\") " Jul 21 22:01:35.588176 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:35.588093 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wzrt\" (UniqueName: \"kubernetes.io/projected/deed685d-7022-4a4e-8fa2-cc38091952ee-kube-api-access-7wzrt\") pod \"deed685d-7022-4a4e-8fa2-cc38091952ee\" (UID: \"deed685d-7022-4a4e-8fa2-cc38091952ee\") " Jul 21 22:01:35.590128 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:35.590094 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deed685d-7022-4a4e-8fa2-cc38091952ee-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "deed685d-7022-4a4e-8fa2-cc38091952ee" (UID: "deed685d-7022-4a4e-8fa2-cc38091952ee"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:01:35.590249 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:35.590227 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deed685d-7022-4a4e-8fa2-cc38091952ee-kube-api-access-7wzrt" (OuterVolumeSpecName: "kube-api-access-7wzrt") pod "deed685d-7022-4a4e-8fa2-cc38091952ee" (UID: "deed685d-7022-4a4e-8fa2-cc38091952ee"). InnerVolumeSpecName "kube-api-access-7wzrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:01:35.689705 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:35.689680 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/deed685d-7022-4a4e-8fa2-cc38091952ee-webhook-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:01:35.689705 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:35.689702 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wzrt\" (UniqueName: \"kubernetes.io/projected/deed685d-7022-4a4e-8fa2-cc38091952ee-kube-api-access-7wzrt\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:01:35.755030 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:35.754995 2580 generic.go:356] "Generic (PLEG): container finished" podID="deed685d-7022-4a4e-8fa2-cc38091952ee" containerID="c8c5bb6e67c1b5c562e4b9bb1672ad6052ee49c320b979a8f3f32cd170d16a53" exitCode=0 Jul 21 22:01:35.755212 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:35.755054 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55bc4db677-htrvj" Jul 21 22:01:35.755212 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:35.755080 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55bc4db677-htrvj" event={"ID":"deed685d-7022-4a4e-8fa2-cc38091952ee","Type":"ContainerDied","Data":"c8c5bb6e67c1b5c562e4b9bb1672ad6052ee49c320b979a8f3f32cd170d16a53"} Jul 21 22:01:35.755212 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:35.755168 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55bc4db677-htrvj" event={"ID":"deed685d-7022-4a4e-8fa2-cc38091952ee","Type":"ContainerDied","Data":"d3b412890b2610de12561aaa8fefadec517819b4980ecfd0c39b6434e5576629"} Jul 21 22:01:35.755212 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:35.755197 2580 scope.go:117] "RemoveContainer" containerID="c8c5bb6e67c1b5c562e4b9bb1672ad6052ee49c320b979a8f3f32cd170d16a53" Jul 21 22:01:35.765718 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:35.765701 2580 scope.go:117] "RemoveContainer" containerID="c8c5bb6e67c1b5c562e4b9bb1672ad6052ee49c320b979a8f3f32cd170d16a53" Jul 21 22:01:35.765959 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:01:35.765941 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8c5bb6e67c1b5c562e4b9bb1672ad6052ee49c320b979a8f3f32cd170d16a53\": container with ID starting with c8c5bb6e67c1b5c562e4b9bb1672ad6052ee49c320b979a8f3f32cd170d16a53 not found: ID does not exist" containerID="c8c5bb6e67c1b5c562e4b9bb1672ad6052ee49c320b979a8f3f32cd170d16a53" Jul 21 22:01:35.766023 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:35.765969 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c5bb6e67c1b5c562e4b9bb1672ad6052ee49c320b979a8f3f32cd170d16a53"} err="failed to get container status \"c8c5bb6e67c1b5c562e4b9bb1672ad6052ee49c320b979a8f3f32cd170d16a53\": rpc error: code = NotFound desc = could not find container \"c8c5bb6e67c1b5c562e4b9bb1672ad6052ee49c320b979a8f3f32cd170d16a53\": container with ID starting with c8c5bb6e67c1b5c562e4b9bb1672ad6052ee49c320b979a8f3f32cd170d16a53 not found: ID does not exist" Jul 21 22:01:35.792788 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:35.792753 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55bc4db677-htrvj"] Jul 21 22:01:35.805107 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:35.805083 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55bc4db677-htrvj"] Jul 21 22:01:36.780490 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:01:36.780451 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deed685d-7022-4a4e-8fa2-cc38091952ee" path="/var/lib/kubelet/pods/deed685d-7022-4a4e-8fa2-cc38091952ee/volumes" Jul 21 22:02:36.726955 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:36.726926 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55bc4db677-msj7x"] Jul 21 22:02:36.727407 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:36.727365 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc169b0d-34ca-4a21-84c3-16fe6de2174a" containerName="authorino" Jul 21 22:02:36.727457 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:36.727409 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc169b0d-34ca-4a21-84c3-16fe6de2174a" containerName="authorino" Jul 21 22:02:36.727457 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:36.727425 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deed685d-7022-4a4e-8fa2-cc38091952ee" containerName="manager" Jul 21 22:02:36.727457 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:36.727430 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="deed685d-7022-4a4e-8fa2-cc38091952ee" containerName="manager" Jul 21 22:02:36.727553 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:36.727515 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="deed685d-7022-4a4e-8fa2-cc38091952ee" containerName="manager" Jul 21 22:02:36.727553 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:36.727526 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc169b0d-34ca-4a21-84c3-16fe6de2174a" containerName="authorino" Jul 21 22:02:36.730492 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:36.730477 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55bc4db677-msj7x" Jul 21 22:02:36.738760 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:36.738742 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 22:02:36.738858 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:36.738744 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 22:02:36.738858 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:36.738744 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-f44sr\"" Jul 21 22:02:36.747287 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:36.747266 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55bc4db677-msj7x"] Jul 21 22:02:36.773040 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:36.773002 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0d66445-1800-44a9-b2f9-5cf82e02efca-webhook-cert\") pod \"maas-controller-55bc4db677-msj7x\" (UID: \"d0d66445-1800-44a9-b2f9-5cf82e02efca\") " pod="opendatahub/maas-controller-55bc4db677-msj7x" Jul 21 22:02:36.773158 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:36.773076 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx65w\" (UniqueName: \"kubernetes.io/projected/d0d66445-1800-44a9-b2f9-5cf82e02efca-kube-api-access-dx65w\") pod \"maas-controller-55bc4db677-msj7x\" (UID: \"d0d66445-1800-44a9-b2f9-5cf82e02efca\") " pod="opendatahub/maas-controller-55bc4db677-msj7x" Jul 21 22:02:36.874119 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:36.874091 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0d66445-1800-44a9-b2f9-5cf82e02efca-webhook-cert\") pod \"maas-controller-55bc4db677-msj7x\" (UID: \"d0d66445-1800-44a9-b2f9-5cf82e02efca\") " pod="opendatahub/maas-controller-55bc4db677-msj7x" Jul 21 22:02:36.874251 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:36.874155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx65w\" (UniqueName: \"kubernetes.io/projected/d0d66445-1800-44a9-b2f9-5cf82e02efca-kube-api-access-dx65w\") pod \"maas-controller-55bc4db677-msj7x\" (UID: \"d0d66445-1800-44a9-b2f9-5cf82e02efca\") " pod="opendatahub/maas-controller-55bc4db677-msj7x" Jul 21 22:02:36.876710 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:36.876688 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0d66445-1800-44a9-b2f9-5cf82e02efca-webhook-cert\") pod \"maas-controller-55bc4db677-msj7x\" (UID: \"d0d66445-1800-44a9-b2f9-5cf82e02efca\") " pod="opendatahub/maas-controller-55bc4db677-msj7x" Jul 21 22:02:36.886802 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:36.886777 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx65w\" (UniqueName: \"kubernetes.io/projected/d0d66445-1800-44a9-b2f9-5cf82e02efca-kube-api-access-dx65w\") pod \"maas-controller-55bc4db677-msj7x\" (UID: \"d0d66445-1800-44a9-b2f9-5cf82e02efca\") " pod="opendatahub/maas-controller-55bc4db677-msj7x" Jul 21 22:02:37.040743 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:37.040711 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55bc4db677-msj7x" Jul 21 22:02:37.184444 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:37.184421 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55bc4db677-msj7x"] Jul 21 22:02:37.186929 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:02:37.186900 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d66445_1800_44a9_b2f9_5cf82e02efca.slice/crio-3f1924d393ddf41bd67b0518995b5987d14c47c5b15962792b511a25da30d269 WatchSource:0}: Error finding container 3f1924d393ddf41bd67b0518995b5987d14c47c5b15962792b511a25da30d269: Status 404 returned error can't find the container with id 3f1924d393ddf41bd67b0518995b5987d14c47c5b15962792b511a25da30d269 Jul 21 22:02:38.077077 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:38.077036 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55bc4db677-msj7x" event={"ID":"d0d66445-1800-44a9-b2f9-5cf82e02efca","Type":"ContainerStarted","Data":"7998899f0cb1ce425839bb2ac392ae222cce6fecd26281d417ac062e15acf956"} Jul 21 22:02:38.077573 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:38.077140 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55bc4db677-msj7x" Jul 21 22:02:38.077573 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:38.077168 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55bc4db677-msj7x" event={"ID":"d0d66445-1800-44a9-b2f9-5cf82e02efca","Type":"ContainerStarted","Data":"3f1924d393ddf41bd67b0518995b5987d14c47c5b15962792b511a25da30d269"} Jul 21 22:02:49.085251 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:49.085215 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55bc4db677-msj7x" Jul 21 22:02:49.126892 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:49.126842 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55bc4db677-msj7x" podStartSLOduration=12.640530352 podStartE2EDuration="13.126825494s" totalImagesPullingTime="486.295142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:02:38.135221951 +0000 UTC m=+835.898456212" watchObservedRunningTime="2026-07-21 22:02:49.126825494 +0000 UTC m=+846.890059782" Jul 21 22:02:53.893848 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:53.893816 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55bc4db677-msj7x"] Jul 21 22:02:53.894319 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:53.893993 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-55bc4db677-msj7x" podUID="d0d66445-1800-44a9-b2f9-5cf82e02efca" containerName="manager" containerID="cri-o://7998899f0cb1ce425839bb2ac392ae222cce6fecd26281d417ac062e15acf956" gracePeriod=30 Jul 21 22:02:53.950306 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:02:53.950273 2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d66445_1800_44a9_b2f9_5cf82e02efca.slice/crio-conmon-7998899f0cb1ce425839bb2ac392ae222cce6fecd26281d417ac062e15acf956.scope\": RecentStats: unable to find data in memory cache]" Jul 21 22:02:54.152031 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:54.151974 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55bc4db677-msj7x" Jul 21 22:02:54.159353 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:54.159327 2580 generic.go:356] "Generic (PLEG): container finished" podID="d0d66445-1800-44a9-b2f9-5cf82e02efca" containerID="7998899f0cb1ce425839bb2ac392ae222cce6fecd26281d417ac062e15acf956" exitCode=0 Jul 21 22:02:54.159523 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:54.159363 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55bc4db677-msj7x" event={"ID":"d0d66445-1800-44a9-b2f9-5cf82e02efca","Type":"ContainerDied","Data":"7998899f0cb1ce425839bb2ac392ae222cce6fecd26281d417ac062e15acf956"} Jul 21 22:02:54.159523 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:54.159393 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55bc4db677-msj7x" Jul 21 22:02:54.159523 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:54.159433 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55bc4db677-msj7x" event={"ID":"d0d66445-1800-44a9-b2f9-5cf82e02efca","Type":"ContainerDied","Data":"3f1924d393ddf41bd67b0518995b5987d14c47c5b15962792b511a25da30d269"} Jul 21 22:02:54.159523 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:54.159457 2580 scope.go:117] "RemoveContainer" containerID="7998899f0cb1ce425839bb2ac392ae222cce6fecd26281d417ac062e15acf956" Jul 21 22:02:54.167765 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:54.167750 2580 scope.go:117] "RemoveContainer" containerID="7998899f0cb1ce425839bb2ac392ae222cce6fecd26281d417ac062e15acf956" Jul 21 22:02:54.168054 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:02:54.168032 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7998899f0cb1ce425839bb2ac392ae222cce6fecd26281d417ac062e15acf956\": container with ID starting with 7998899f0cb1ce425839bb2ac392ae222cce6fecd26281d417ac062e15acf956 not found: ID does not exist" containerID="7998899f0cb1ce425839bb2ac392ae222cce6fecd26281d417ac062e15acf956" Jul 21 22:02:54.168122 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:54.168068 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7998899f0cb1ce425839bb2ac392ae222cce6fecd26281d417ac062e15acf956"} err="failed to get container status \"7998899f0cb1ce425839bb2ac392ae222cce6fecd26281d417ac062e15acf956\": rpc error: code = NotFound desc = could not find container \"7998899f0cb1ce425839bb2ac392ae222cce6fecd26281d417ac062e15acf956\": container with ID starting with 7998899f0cb1ce425839bb2ac392ae222cce6fecd26281d417ac062e15acf956 not found: ID does not exist" Jul 21 22:02:54.346321 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:54.346291 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx65w\" (UniqueName: \"kubernetes.io/projected/d0d66445-1800-44a9-b2f9-5cf82e02efca-kube-api-access-dx65w\") pod \"d0d66445-1800-44a9-b2f9-5cf82e02efca\" (UID: \"d0d66445-1800-44a9-b2f9-5cf82e02efca\") " Jul 21 22:02:54.346550 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:54.346410 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0d66445-1800-44a9-b2f9-5cf82e02efca-webhook-cert\") pod \"d0d66445-1800-44a9-b2f9-5cf82e02efca\" (UID: \"d0d66445-1800-44a9-b2f9-5cf82e02efca\") " Jul 21 22:02:54.348532 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:54.348499 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d66445-1800-44a9-b2f9-5cf82e02efca-kube-api-access-dx65w" (OuterVolumeSpecName: "kube-api-access-dx65w") pod "d0d66445-1800-44a9-b2f9-5cf82e02efca" (UID: "d0d66445-1800-44a9-b2f9-5cf82e02efca"). InnerVolumeSpecName "kube-api-access-dx65w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:02:54.348667 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:54.348583 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d66445-1800-44a9-b2f9-5cf82e02efca-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d0d66445-1800-44a9-b2f9-5cf82e02efca" (UID: "d0d66445-1800-44a9-b2f9-5cf82e02efca"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:02:54.447831 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:54.447761 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0d66445-1800-44a9-b2f9-5cf82e02efca-webhook-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:02:54.447831 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:54.447789 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx65w\" (UniqueName: \"kubernetes.io/projected/d0d66445-1800-44a9-b2f9-5cf82e02efca-kube-api-access-dx65w\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:02:54.505236 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:54.505208 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55bc4db677-msj7x"] Jul 21 22:02:54.515850 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:54.515829 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55bc4db677-msj7x"] Jul 21 22:02:54.780891 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:54.780862 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0d66445-1800-44a9-b2f9-5cf82e02efca" path="/var/lib/kubelet/pods/d0d66445-1800-44a9-b2f9-5cf82e02efca/volumes" Jul 21 22:02:55.470006 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:55.469977 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55bc4db677-q2m5g"] Jul 21 22:02:55.470676 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:55.470484 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d66445-1800-44a9-b2f9-5cf82e02efca" containerName="manager" Jul 21 22:02:55.470676 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:55.470497 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d66445-1800-44a9-b2f9-5cf82e02efca" containerName="manager" Jul 21 22:02:55.470676 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:55.470585 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d66445-1800-44a9-b2f9-5cf82e02efca" containerName="manager" Jul 21 22:02:55.475397 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:55.475360 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55bc4db677-q2m5g" Jul 21 22:02:55.489999 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:55.489972 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-f44sr\"" Jul 21 22:02:55.490087 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:55.490054 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 22:02:55.490124 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:55.490064 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 22:02:55.503023 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:55.502997 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55bc4db677-q2m5g"] Jul 21 22:02:55.554904 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:55.554883 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vldjk\" (UniqueName: \"kubernetes.io/projected/f26946dd-d05a-445e-878a-a96b65f407eb-kube-api-access-vldjk\") pod \"maas-controller-55bc4db677-q2m5g\" (UID: \"f26946dd-d05a-445e-878a-a96b65f407eb\") " pod="opendatahub/maas-controller-55bc4db677-q2m5g" Jul 21 22:02:55.555037 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:55.554976 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f26946dd-d05a-445e-878a-a96b65f407eb-webhook-cert\") pod \"maas-controller-55bc4db677-q2m5g\" (UID: \"f26946dd-d05a-445e-878a-a96b65f407eb\") " pod="opendatahub/maas-controller-55bc4db677-q2m5g" Jul 21 22:02:55.655465 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:55.655434 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vldjk\" (UniqueName: \"kubernetes.io/projected/f26946dd-d05a-445e-878a-a96b65f407eb-kube-api-access-vldjk\") pod \"maas-controller-55bc4db677-q2m5g\" (UID: \"f26946dd-d05a-445e-878a-a96b65f407eb\") " pod="opendatahub/maas-controller-55bc4db677-q2m5g" Jul 21 22:02:55.655594 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:55.655525 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f26946dd-d05a-445e-878a-a96b65f407eb-webhook-cert\") pod \"maas-controller-55bc4db677-q2m5g\" (UID: \"f26946dd-d05a-445e-878a-a96b65f407eb\") " pod="opendatahub/maas-controller-55bc4db677-q2m5g" Jul 21 22:02:55.658057 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:55.658034 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f26946dd-d05a-445e-878a-a96b65f407eb-webhook-cert\") pod \"maas-controller-55bc4db677-q2m5g\" (UID: \"f26946dd-d05a-445e-878a-a96b65f407eb\") " pod="opendatahub/maas-controller-55bc4db677-q2m5g" Jul 21 22:02:55.670757 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:55.670737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vldjk\" (UniqueName: \"kubernetes.io/projected/f26946dd-d05a-445e-878a-a96b65f407eb-kube-api-access-vldjk\") pod \"maas-controller-55bc4db677-q2m5g\" (UID: \"f26946dd-d05a-445e-878a-a96b65f407eb\") " pod="opendatahub/maas-controller-55bc4db677-q2m5g" Jul 21 22:02:55.785874 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:55.785850 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55bc4db677-q2m5g" Jul 21 22:02:55.935750 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:55.935714 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55bc4db677-q2m5g"] Jul 21 22:02:55.941111 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:02:55.941087 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26946dd_d05a_445e_878a_a96b65f407eb.slice/crio-269ffd99647b407dba2a7e20ca6ebc8add449d1f94bb78460dd42c93682c5732 WatchSource:0}: Error finding container 269ffd99647b407dba2a7e20ca6ebc8add449d1f94bb78460dd42c93682c5732: Status 404 returned error can't find the container with id 269ffd99647b407dba2a7e20ca6ebc8add449d1f94bb78460dd42c93682c5732 Jul 21 22:02:56.172463 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:56.172350 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55bc4db677-q2m5g" event={"ID":"f26946dd-d05a-445e-878a-a96b65f407eb","Type":"ContainerStarted","Data":"269ffd99647b407dba2a7e20ca6ebc8add449d1f94bb78460dd42c93682c5732"} Jul 21 22:02:57.179389 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:57.179337 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55bc4db677-q2m5g" event={"ID":"f26946dd-d05a-445e-878a-a96b65f407eb","Type":"ContainerStarted","Data":"3e46556916ab1aee263e8bfea9b601768658217655bfee4383772788433b48f8"} Jul 21 22:02:57.179825 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:57.179459 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55bc4db677-q2m5g" Jul 21 22:02:57.221365 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:02:57.221317 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55bc4db677-q2m5g" podStartSLOduration=1.799786 podStartE2EDuration="2.221303808s" totalImagesPullingTime="421.517808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:02:57.220181087 +0000 UTC m=+854.983415347" watchObservedRunningTime="2026-07-21 22:02:57.221303808 +0000 UTC m=+854.984538068" Jul 21 22:03:08.186719 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:03:08.186685 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55bc4db677-q2m5g" Jul 21 22:07:46.551151 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:07:46.551119 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf"] Jul 21 22:07:46.551940 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:07:46.551315 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf" podUID="aafea688-097a-4bf5-bd34-011da016e6c9" containerName="manager" containerID="cri-o://83f419730d3cfa270497f7bd9701035da0ec3bd1bcf2f5ba9ca441b98dd69de0" gracePeriod=10 Jul 21 22:07:48.686934 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:07:48.686905 2580 generic.go:356] "Generic (PLEG): container finished" podID="aafea688-097a-4bf5-bd34-011da016e6c9" containerID="83f419730d3cfa270497f7bd9701035da0ec3bd1bcf2f5ba9ca441b98dd69de0" exitCode=0 Jul 21 22:07:48.687247 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:07:48.686984 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf" event={"ID":"aafea688-097a-4bf5-bd34-011da016e6c9","Type":"ContainerDied","Data":"83f419730d3cfa270497f7bd9701035da0ec3bd1bcf2f5ba9ca441b98dd69de0"} Jul 21 22:07:49.304891 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:07:49.304868 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf" Jul 21 22:07:49.350013 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:07:49.349984 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcq5g\" (UniqueName: \"kubernetes.io/projected/aafea688-097a-4bf5-bd34-011da016e6c9-kube-api-access-rcq5g\") pod \"aafea688-097a-4bf5-bd34-011da016e6c9\" (UID: \"aafea688-097a-4bf5-bd34-011da016e6c9\") " Jul 21 22:07:49.350152 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:07:49.350073 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aafea688-097a-4bf5-bd34-011da016e6c9-extensions-socket-volume\") pod \"aafea688-097a-4bf5-bd34-011da016e6c9\" (UID: \"aafea688-097a-4bf5-bd34-011da016e6c9\") " Jul 21 22:07:49.350477 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:07:49.350448 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafea688-097a-4bf5-bd34-011da016e6c9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "aafea688-097a-4bf5-bd34-011da016e6c9" (UID: "aafea688-097a-4bf5-bd34-011da016e6c9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:07:49.352260 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:07:49.352233 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafea688-097a-4bf5-bd34-011da016e6c9-kube-api-access-rcq5g" (OuterVolumeSpecName: "kube-api-access-rcq5g") pod "aafea688-097a-4bf5-bd34-011da016e6c9" (UID: "aafea688-097a-4bf5-bd34-011da016e6c9"). InnerVolumeSpecName "kube-api-access-rcq5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:07:49.451522 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:07:49.451455 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcq5g\" (UniqueName: \"kubernetes.io/projected/aafea688-097a-4bf5-bd34-011da016e6c9-kube-api-access-rcq5g\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:07:49.451522 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:07:49.451482 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aafea688-097a-4bf5-bd34-011da016e6c9-extensions-socket-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:07:49.693675 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:07:49.693648 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf" Jul 21 22:07:49.693675 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:07:49.693666 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf" event={"ID":"aafea688-097a-4bf5-bd34-011da016e6c9","Type":"ContainerDied","Data":"b3b9afdd8d98c95070a36a31515df63fae106529caaf700259db4ee765692eff"} Jul 21 22:07:49.694126 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:07:49.693728 2580 scope.go:117] "RemoveContainer" containerID="83f419730d3cfa270497f7bd9701035da0ec3bd1bcf2f5ba9ca441b98dd69de0" Jul 21 22:07:49.721196 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:07:49.721165 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf"] Jul 21 22:07:49.729253 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:07:49.729223 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-f5lbf"] Jul 21 22:07:50.779111 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:07:50.779071 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aafea688-097a-4bf5-bd34-011da016e6c9" path="/var/lib/kubelet/pods/aafea688-097a-4bf5-bd34-011da016e6c9/volumes" Jul 21 22:08:47.772785 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:47.772749 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2p49"] Jul 21 22:08:47.773398 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:47.773357 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aafea688-097a-4bf5-bd34-011da016e6c9" containerName="manager" Jul 21 22:08:47.773398 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:47.773395 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aafea688-097a-4bf5-bd34-011da016e6c9" containerName="manager" Jul 21 22:08:47.773552 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:47.773544 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aafea688-097a-4bf5-bd34-011da016e6c9" containerName="manager" Jul 21 22:08:47.778066 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:47.778039 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2p49" Jul 21 22:08:47.781594 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:47.781575 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tsvj5\"" Jul 21 22:08:47.793637 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:47.793616 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2p49"] Jul 21 22:08:47.860927 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:47.860891 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qgt7\" (UniqueName: \"kubernetes.io/projected/1b1ddb27-37c1-4f88-907f-bc569c6c971e-kube-api-access-6qgt7\") pod \"kuadrant-operator-controller-manager-86878c769f-d2p49\" (UID: \"1b1ddb27-37c1-4f88-907f-bc569c6c971e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2p49" Jul 21 22:08:47.861099 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:47.861056 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b1ddb27-37c1-4f88-907f-bc569c6c971e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-d2p49\" (UID: \"1b1ddb27-37c1-4f88-907f-bc569c6c971e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2p49" Jul 21 22:08:47.962770 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:47.962739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qgt7\" (UniqueName: \"kubernetes.io/projected/1b1ddb27-37c1-4f88-907f-bc569c6c971e-kube-api-access-6qgt7\") pod \"kuadrant-operator-controller-manager-86878c769f-d2p49\" (UID: \"1b1ddb27-37c1-4f88-907f-bc569c6c971e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2p49" Jul 21 22:08:47.962903 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:47.962786 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b1ddb27-37c1-4f88-907f-bc569c6c971e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-d2p49\" (UID: \"1b1ddb27-37c1-4f88-907f-bc569c6c971e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2p49" Jul 21 22:08:47.963160 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:47.963143 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b1ddb27-37c1-4f88-907f-bc569c6c971e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-d2p49\" (UID: \"1b1ddb27-37c1-4f88-907f-bc569c6c971e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2p49" Jul 21 22:08:47.973056 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:47.973031 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qgt7\" (UniqueName: \"kubernetes.io/projected/1b1ddb27-37c1-4f88-907f-bc569c6c971e-kube-api-access-6qgt7\") pod \"kuadrant-operator-controller-manager-86878c769f-d2p49\" (UID: \"1b1ddb27-37c1-4f88-907f-bc569c6c971e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2p49" Jul 21 22:08:48.088413 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:48.088312 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2p49" Jul 21 22:08:48.232666 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:48.232641 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2p49"] Jul 21 22:08:48.235271 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:08:48.235246 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1ddb27_37c1_4f88_907f_bc569c6c971e.slice/crio-484e989e5eabad7f434889e0309fc4d3f57e833fa27e6ddf36e785a94462ee95 WatchSource:0}: Error finding container 484e989e5eabad7f434889e0309fc4d3f57e833fa27e6ddf36e785a94462ee95: Status 404 returned error can't find the container with id 484e989e5eabad7f434889e0309fc4d3f57e833fa27e6ddf36e785a94462ee95 Jul 21 22:08:48.985250 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:48.985215 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2p49" event={"ID":"1b1ddb27-37c1-4f88-907f-bc569c6c971e","Type":"ContainerStarted","Data":"883f98213960e8e3b365b89f00a89fe7e96ddf39e36955ff92fd3d83d58c3a46"} Jul 21 22:08:48.985669 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:48.985277 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2p49" event={"ID":"1b1ddb27-37c1-4f88-907f-bc569c6c971e","Type":"ContainerStarted","Data":"484e989e5eabad7f434889e0309fc4d3f57e833fa27e6ddf36e785a94462ee95"} Jul 21 22:08:48.985669 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:48.985307 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2p49" Jul 21 22:08:49.020928 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:49.020880 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2p49" podStartSLOduration=2.020864243 podStartE2EDuration="2.020864243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:08:49.016985965 +0000 UTC m=+1206.780220228" watchObservedRunningTime="2026-07-21 22:08:49.020864243 +0000 UTC m=+1206.784098504" Jul 21 22:08:59.993903 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:08:59.993873 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d2p49" Jul 21 22:11:49.778562 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.778516 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv"] Jul 21 22:11:49.783974 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.783932 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:49.786845 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.786822 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c461abb7-kserve-self-signed-certs\"" Jul 21 22:11:49.786988 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.786868 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv"] Jul 21 22:11:49.802187 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.802157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-home\") pod \"e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:49.802315 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.802194 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-tmp-dir\") pod \"e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:49.802315 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.802217 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-model-cache\") pod \"e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:49.802315 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.802269 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw2sk\" (UniqueName: \"kubernetes.io/projected/bb706077-c36c-48e8-b664-0b0e8d4862a3-kube-api-access-fw2sk\") pod \"e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:49.802455 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.802357 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-dshm\") pod \"e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:49.802455 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.802421 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb706077-c36c-48e8-b664-0b0e8d4862a3-tls-certs\") pod \"e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:49.903504 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.903469 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-home\") pod \"e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:49.903719 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.903511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-tmp-dir\") pod \"e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:49.903719 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.903534 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-model-cache\") pod \"e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:49.903719 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.903564 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw2sk\" (UniqueName: \"kubernetes.io/projected/bb706077-c36c-48e8-b664-0b0e8d4862a3-kube-api-access-fw2sk\") pod \"e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:49.903719 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.903649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-dshm\") pod \"e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:49.903719 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.903681 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb706077-c36c-48e8-b664-0b0e8d4862a3-tls-certs\") pod \"e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:49.903993 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.903947 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-tmp-dir\") pod \"e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:49.903993 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.903965 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-home\") pod \"e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:49.904106 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.904057 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-model-cache\") pod \"e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:49.906113 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.906090 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-dshm\") pod \"e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:49.906268 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.906252 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb706077-c36c-48e8-b664-0b0e8d4862a3-tls-certs\") pod \"e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:49.912102 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:49.912081 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw2sk\" (UniqueName: \"kubernetes.io/projected/bb706077-c36c-48e8-b664-0b0e8d4862a3-kube-api-access-fw2sk\") pod \"e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:50.094144 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:50.094061 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:11:50.227044 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:50.227021 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv"] Jul 21 22:11:50.229604 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:11:50.229575 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb706077_c36c_48e8_b664_0b0e8d4862a3.slice/crio-91474718d083fa7ea763be071fae62a65a703ee29fd3ab31665520cf00a80b4d WatchSource:0}: Error finding container 91474718d083fa7ea763be071fae62a65a703ee29fd3ab31665520cf00a80b4d: Status 404 returned error can't find the container with id 91474718d083fa7ea763be071fae62a65a703ee29fd3ab31665520cf00a80b4d Jul 21 22:11:50.938945 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:50.938916 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" event={"ID":"bb706077-c36c-48e8-b664-0b0e8d4862a3","Type":"ContainerStarted","Data":"b480239992573d4a14df1b235efa9724465943f16f3b9392913f71871cbf2d5b"} Jul 21 22:11:50.939325 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:50.938953 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" event={"ID":"bb706077-c36c-48e8-b664-0b0e8d4862a3","Type":"ContainerStarted","Data":"91474718d083fa7ea763be071fae62a65a703ee29fd3ab31665520cf00a80b4d"} Jul 21 22:11:50.957570 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:11:50.957524 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" podStartSLOduration=1.957509486 podStartE2EDuration="1.957509486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:11:50.956169478 +0000 UTC m=+1388.719403756" watchObservedRunningTime="2026-07-21 22:11:50.957509486 +0000 UTC m=+1388.720743747" Jul 21 22:12:00.095175 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:00.095134 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:12:00.095175 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:00.095182 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:12:00.107682 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:00.107655 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:12:01.008251 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:01.008216 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:12:13.497873 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:13.497833 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv"] Jul 21 22:12:28.523938 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.523878 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" podUID="bb706077-c36c-48e8-b664-0b0e8d4862a3" containerName="main" containerID="cri-o://b480239992573d4a14df1b235efa9724465943f16f3b9392913f71871cbf2d5b" gracePeriod=45 Jul 21 22:12:28.784859 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.784792 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:12:28.791703 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.791681 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-home\") pod \"bb706077-c36c-48e8-b664-0b0e8d4862a3\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " Jul 21 22:12:28.791787 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.791728 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-dshm\") pod \"bb706077-c36c-48e8-b664-0b0e8d4862a3\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " Jul 21 22:12:28.791787 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.791771 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-model-cache\") pod \"bb706077-c36c-48e8-b664-0b0e8d4862a3\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " Jul 21 22:12:28.791858 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.791801 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw2sk\" (UniqueName: \"kubernetes.io/projected/bb706077-c36c-48e8-b664-0b0e8d4862a3-kube-api-access-fw2sk\") pod \"bb706077-c36c-48e8-b664-0b0e8d4862a3\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " Jul 21 22:12:28.791900 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.791880 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb706077-c36c-48e8-b664-0b0e8d4862a3-tls-certs\") pod \"bb706077-c36c-48e8-b664-0b0e8d4862a3\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " Jul 21 22:12:28.791947 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.791912 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-tmp-dir\") pod \"bb706077-c36c-48e8-b664-0b0e8d4862a3\" (UID: \"bb706077-c36c-48e8-b664-0b0e8d4862a3\") " Jul 21 22:12:28.791947 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.791918 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-home" (OuterVolumeSpecName: "home") pod "bb706077-c36c-48e8-b664-0b0e8d4862a3" (UID: "bb706077-c36c-48e8-b664-0b0e8d4862a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:12:28.792082 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.792038 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb706077-c36c-48e8-b664-0b0e8d4862a3" (UID: "bb706077-c36c-48e8-b664-0b0e8d4862a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:12:28.792191 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.792169 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb706077-c36c-48e8-b664-0b0e8d4862a3" (UID: "bb706077-c36c-48e8-b664-0b0e8d4862a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:12:28.792316 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.792299 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-tmp-dir\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:12:28.792392 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.792321 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-home\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:12:28.792392 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.792335 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-model-cache\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:12:28.794174 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.794154 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb706077-c36c-48e8-b664-0b0e8d4862a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb706077-c36c-48e8-b664-0b0e8d4862a3" (UID: "bb706077-c36c-48e8-b664-0b0e8d4862a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:12:28.794174 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.794161 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-dshm" (OuterVolumeSpecName: "dshm") pod "bb706077-c36c-48e8-b664-0b0e8d4862a3" (UID: "bb706077-c36c-48e8-b664-0b0e8d4862a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:12:28.794262 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.794181 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb706077-c36c-48e8-b664-0b0e8d4862a3-kube-api-access-fw2sk" (OuterVolumeSpecName: "kube-api-access-fw2sk") pod "bb706077-c36c-48e8-b664-0b0e8d4862a3" (UID: "bb706077-c36c-48e8-b664-0b0e8d4862a3"). InnerVolumeSpecName "kube-api-access-fw2sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:12:28.893494 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.893465 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw2sk\" (UniqueName: \"kubernetes.io/projected/bb706077-c36c-48e8-b664-0b0e8d4862a3-kube-api-access-fw2sk\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:12:28.893494 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.893500 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb706077-c36c-48e8-b664-0b0e8d4862a3-tls-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:12:28.893699 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:28.893511 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb706077-c36c-48e8-b664-0b0e8d4862a3-dshm\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:12:29.140137 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:29.140047 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb706077-c36c-48e8-b664-0b0e8d4862a3" containerID="b480239992573d4a14df1b235efa9724465943f16f3b9392913f71871cbf2d5b" exitCode=0 Jul 21 22:12:29.140137 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:29.140087 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" event={"ID":"bb706077-c36c-48e8-b664-0b0e8d4862a3","Type":"ContainerDied","Data":"b480239992573d4a14df1b235efa9724465943f16f3b9392913f71871cbf2d5b"} Jul 21 22:12:29.140137 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:29.140093 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" Jul 21 22:12:29.140137 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:29.140121 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv" event={"ID":"bb706077-c36c-48e8-b664-0b0e8d4862a3","Type":"ContainerDied","Data":"91474718d083fa7ea763be071fae62a65a703ee29fd3ab31665520cf00a80b4d"} Jul 21 22:12:29.140458 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:29.140143 2580 scope.go:117] "RemoveContainer" containerID="b480239992573d4a14df1b235efa9724465943f16f3b9392913f71871cbf2d5b" Jul 21 22:12:29.149767 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:29.149748 2580 scope.go:117] "RemoveContainer" containerID="b480239992573d4a14df1b235efa9724465943f16f3b9392913f71871cbf2d5b" Jul 21 22:12:29.150004 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:12:29.149984 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b480239992573d4a14df1b235efa9724465943f16f3b9392913f71871cbf2d5b\": container with ID starting with b480239992573d4a14df1b235efa9724465943f16f3b9392913f71871cbf2d5b not found: ID does not exist" containerID="b480239992573d4a14df1b235efa9724465943f16f3b9392913f71871cbf2d5b" Jul 21 22:12:29.150049 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:29.150013 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b480239992573d4a14df1b235efa9724465943f16f3b9392913f71871cbf2d5b"} err="failed to get container status \"b480239992573d4a14df1b235efa9724465943f16f3b9392913f71871cbf2d5b\": rpc error: code = NotFound desc = could not find container \"b480239992573d4a14df1b235efa9724465943f16f3b9392913f71871cbf2d5b\": container with ID starting with b480239992573d4a14df1b235efa9724465943f16f3b9392913f71871cbf2d5b not found: ID does not exist" Jul 21 22:12:29.161751 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:29.161723 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv"] Jul 21 22:12:29.165720 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:29.165698 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c461abb7-kserve-75f8b48695-hlbjv"] Jul 21 22:12:30.780300 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:12:30.780270 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb706077-c36c-48e8-b664-0b0e8d4862a3" path="/var/lib/kubelet/pods/bb706077-c36c-48e8-b664-0b0e8d4862a3/volumes" Jul 21 22:15:53.639729 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.639649 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm"] Jul 21 22:15:53.640170 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.640142 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb706077-c36c-48e8-b664-0b0e8d4862a3" containerName="main" Jul 21 22:15:53.640170 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.640156 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb706077-c36c-48e8-b664-0b0e8d4862a3" containerName="main" Jul 21 22:15:53.640241 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.640232 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb706077-c36c-48e8-b664-0b0e8d4862a3" containerName="main" Jul 21 22:15:53.643204 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.643186 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.645814 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.645793 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-678339c2-openshift-default-dockercfg-wztr8\"" Jul 21 22:15:53.654873 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.654842 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm"] Jul 21 22:15:53.750180 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.750141 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-istio-envoy\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.750347 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.750194 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-istio-data\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.750347 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.750216 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-credential-socket\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.750347 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.750236 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f220aa-6950-47d0-8574-e6752ab22716-istiod-ca-cert\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.750347 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.750251 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-workload-socket\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.750347 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.750271 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djm8d\" (UniqueName: \"kubernetes.io/projected/22f220aa-6950-47d0-8574-e6752ab22716-kube-api-access-djm8d\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.750347 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.750321 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f220aa-6950-47d0-8574-e6752ab22716-istio-podinfo\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.750599 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.750425 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-workload-certs\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.750599 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.750504 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f220aa-6950-47d0-8574-e6752ab22716-istio-token\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.851461 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.851432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-workload-certs\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.851668 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.851494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f220aa-6950-47d0-8574-e6752ab22716-istio-token\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.851668 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.851521 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-istio-envoy\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.851668 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.851561 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-istio-data\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.851668 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.851581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-credential-socket\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.851668 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.851599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f220aa-6950-47d0-8574-e6752ab22716-istiod-ca-cert\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.851970 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.851687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-workload-socket\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.851970 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.851727 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djm8d\" (UniqueName: \"kubernetes.io/projected/22f220aa-6950-47d0-8574-e6752ab22716-kube-api-access-djm8d\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.851970 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.851761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f220aa-6950-47d0-8574-e6752ab22716-istio-podinfo\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.851970 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.851895 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-workload-certs\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.851970 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.851942 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-credential-socket\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.852243 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.852091 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-workload-socket\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.852307 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.852285 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-istio-data\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.852354 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.852301 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f220aa-6950-47d0-8574-e6752ab22716-istiod-ca-cert\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.853928 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.853910 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-istio-envoy\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.854256 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.854238 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f220aa-6950-47d0-8574-e6752ab22716-istio-podinfo\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.859956 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.859930 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f220aa-6950-47d0-8574-e6752ab22716-istio-token\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.860291 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.860271 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djm8d\" (UniqueName: \"kubernetes.io/projected/22f220aa-6950-47d0-8574-e6752ab22716-kube-api-access-djm8d\") pod \"e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:53.954687 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:53.954602 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:54.103795 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:54.103766 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm"] Jul 21 22:15:54.105647 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:15:54.105621 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f220aa_6950_47d0_8574_e6752ab22716.slice/crio-660e46fb2465feed832753163df715e931b97940eb56fd617433b5c30f04d0ea WatchSource:0}: Error finding container 660e46fb2465feed832753163df715e931b97940eb56fd617433b5c30f04d0ea: Status 404 returned error can't find the container with id 660e46fb2465feed832753163df715e931b97940eb56fd617433b5c30f04d0ea Jul 21 22:15:54.107852 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:54.107810 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:15:54.107924 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:54.107902 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:15:54.107962 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:54.107942 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:15:54.182839 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:54.182761 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" event={"ID":"22f220aa-6950-47d0-8574-e6752ab22716","Type":"ContainerStarted","Data":"7989791097e274036896424638bdd0f88f005f7d79299843780b7152a0a698c7"} Jul 21 22:15:54.182839 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:54.182809 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" event={"ID":"22f220aa-6950-47d0-8574-e6752ab22716","Type":"ContainerStarted","Data":"660e46fb2465feed832753163df715e931b97940eb56fd617433b5c30f04d0ea"} Jul 21 22:15:54.206850 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:54.206751 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" podStartSLOduration=1.206733234 podStartE2EDuration="1.206733234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:15:54.204917659 +0000 UTC m=+1631.968151931" watchObservedRunningTime="2026-07-21 22:15:54.206733234 +0000 UTC m=+1631.969967494" Jul 21 22:15:54.954730 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:54.954697 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:55.959629 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:55.959590 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:56.193871 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:56.193834 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:15:56.194850 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:15:56.194827 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:16:08.257645 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:08.257611 2580 generic.go:356] "Generic (PLEG): container finished" podID="22f220aa-6950-47d0-8574-e6752ab22716" containerID="7989791097e274036896424638bdd0f88f005f7d79299843780b7152a0a698c7" exitCode=0 Jul 21 22:16:08.258050 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:08.257643 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" event={"ID":"22f220aa-6950-47d0-8574-e6752ab22716","Type":"ContainerDied","Data":"7989791097e274036896424638bdd0f88f005f7d79299843780b7152a0a698c7"} Jul 21 22:16:08.258108 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:08.258089 2580 scope.go:117] "RemoveContainer" containerID="7989791097e274036896424638bdd0f88f005f7d79299843780b7152a0a698c7" Jul 21 22:16:08.259080 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:08.259052 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:16:08.259196 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:08.259136 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:16:08.259196 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:08.259184 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:16:09.264603 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:09.264568 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" event={"ID":"22f220aa-6950-47d0-8574-e6752ab22716","Type":"ContainerStarted","Data":"6b7bc9ae178e83c95a37227dfdfc7a9aba1747a2fe662162c238cd1318242753"} Jul 21 22:16:09.955519 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:09.955482 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:16:09.956346 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:09.956320 2580 patch_prober.go:28] interesting pod/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 21 22:16:09.956474 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:09.956367 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" podUID="22f220aa-6950-47d0-8574-e6752ab22716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 21 22:16:10.955751 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:10.955714 2580 patch_prober.go:28] interesting pod/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 21 22:16:10.956148 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:10.955775 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" podUID="22f220aa-6950-47d0-8574-e6752ab22716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 21 22:16:11.194059 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:11.194012 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:16:11.955829 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:11.955796 2580 patch_prober.go:28] interesting pod/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 21 22:16:11.956226 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:11.955854 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" podUID="22f220aa-6950-47d0-8574-e6752ab22716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 21 22:16:12.955158 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:12.955126 2580 patch_prober.go:28] interesting pod/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 21 22:16:12.955327 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:12.955179 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" podUID="22f220aa-6950-47d0-8574-e6752ab22716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 21 22:16:13.959416 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:13.959358 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:16:13.960350 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:13.960335 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:16:36.422315 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:36.422281 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm"] Jul 21 22:16:36.422950 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:36.422562 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" podUID="22f220aa-6950-47d0-8574-e6752ab22716" containerName="istio-proxy" containerID="cri-o://6b7bc9ae178e83c95a37227dfdfc7a9aba1747a2fe662162c238cd1318242753" gracePeriod=30 Jul 21 22:16:37.514316 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.514282 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2"] Jul 21 22:16:37.518047 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.518026 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.521278 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.521261 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-dd422c5b-gw-openshift-default-dockercfg-c5h7g\"" Jul 21 22:16:37.539905 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.539882 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2"] Jul 21 22:16:37.591120 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.591091 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qrx4\" (UniqueName: \"kubernetes.io/projected/41bbcc55-4322-4c00-94d2-2184fad4d62c-kube-api-access-8qrx4\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.591233 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.591127 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-workload-socket\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.591233 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.591148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41bbcc55-4322-4c00-94d2-2184fad4d62c-istiod-ca-cert\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.591310 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.591224 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-data\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.591310 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.591287 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-token\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.591404 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.591357 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-workload-certs\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.591446 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.591405 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-credential-socket\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.591481 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.591452 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-podinfo\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.591515 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.591486 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-envoy\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.692053 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.692016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-data\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.692053 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.692055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-token\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.692328 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.692105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-workload-certs\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.692328 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.692123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-credential-socket\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.692328 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.692150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-podinfo\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.692328 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.692172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-envoy\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.692328 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.692231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qrx4\" (UniqueName: \"kubernetes.io/projected/41bbcc55-4322-4c00-94d2-2184fad4d62c-kube-api-access-8qrx4\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.692328 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.692266 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-workload-socket\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.692328 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.692295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41bbcc55-4322-4c00-94d2-2184fad4d62c-istiod-ca-cert\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.692881 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.692532 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-data\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.692881 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.692560 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-workload-certs\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.692881 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.692744 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-credential-socket\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.693054 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.692914 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-workload-socket\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.693186 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.693166 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41bbcc55-4322-4c00-94d2-2184fad4d62c-istiod-ca-cert\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.695179 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.695152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-envoy\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.695305 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.695288 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-podinfo\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.714328 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.714309 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-token\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.729834 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.729815 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qrx4\" (UniqueName: \"kubernetes.io/projected/41bbcc55-4322-4c00-94d2-2184fad4d62c-kube-api-access-8qrx4\") pod \"e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.828948 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.828865 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:37.990957 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.990934 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2"] Jul 21 22:16:37.993112 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:16:37.993087 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41bbcc55_4322_4c00_94d2_2184fad4d62c.slice/crio-e2901cf14e493d59684d050708c7209adec4eacab2a0679432366783df245690 WatchSource:0}: Error finding container e2901cf14e493d59684d050708c7209adec4eacab2a0679432366783df245690: Status 404 returned error can't find the container with id e2901cf14e493d59684d050708c7209adec4eacab2a0679432366783df245690 Jul 21 22:16:37.995143 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.995108 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:16:37.995234 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.995185 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:16:37.995287 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:37.995229 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:16:38.438424 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:38.438387 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" event={"ID":"41bbcc55-4322-4c00-94d2-2184fad4d62c","Type":"ContainerStarted","Data":"7f4b2ccc6a5cfd9bf60637be8bf76aeb61b362b51bebc424f69e27556771f59a"} Jul 21 22:16:38.438636 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:38.438456 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" event={"ID":"41bbcc55-4322-4c00-94d2-2184fad4d62c","Type":"ContainerStarted","Data":"e2901cf14e493d59684d050708c7209adec4eacab2a0679432366783df245690"} Jul 21 22:16:38.466843 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:38.466797 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" podStartSLOduration=1.466782072 podStartE2EDuration="1.466782072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:16:38.464087951 +0000 UTC m=+1676.227322213" watchObservedRunningTime="2026-07-21 22:16:38.466782072 +0000 UTC m=+1676.230016399" Jul 21 22:16:38.829804 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:38.829777 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:39.834977 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:39.834947 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:40.448702 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:40.448674 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:40.449492 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:40.449474 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:41.676898 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.676872 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:16:41.733901 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.733825 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f220aa-6950-47d0-8574-e6752ab22716-istio-podinfo\") pod \"22f220aa-6950-47d0-8574-e6752ab22716\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " Jul 21 22:16:41.733901 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.733878 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-istio-envoy\") pod \"22f220aa-6950-47d0-8574-e6752ab22716\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " Jul 21 22:16:41.734105 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.733946 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djm8d\" (UniqueName: \"kubernetes.io/projected/22f220aa-6950-47d0-8574-e6752ab22716-kube-api-access-djm8d\") pod \"22f220aa-6950-47d0-8574-e6752ab22716\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " Jul 21 22:16:41.734105 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.733974 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-workload-socket\") pod \"22f220aa-6950-47d0-8574-e6752ab22716\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " Jul 21 22:16:41.734105 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.734002 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-workload-certs\") pod \"22f220aa-6950-47d0-8574-e6752ab22716\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " Jul 21 22:16:41.734105 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.734031 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f220aa-6950-47d0-8574-e6752ab22716-istiod-ca-cert\") pod \"22f220aa-6950-47d0-8574-e6752ab22716\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " Jul 21 22:16:41.734105 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.734060 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-credential-socket\") pod \"22f220aa-6950-47d0-8574-e6752ab22716\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " Jul 21 22:16:41.734105 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.734093 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-istio-data\") pod \"22f220aa-6950-47d0-8574-e6752ab22716\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " Jul 21 22:16:41.734480 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.734164 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f220aa-6950-47d0-8574-e6752ab22716-istio-token\") pod \"22f220aa-6950-47d0-8574-e6752ab22716\" (UID: \"22f220aa-6950-47d0-8574-e6752ab22716\") " Jul 21 22:16:41.734542 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.734478 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "22f220aa-6950-47d0-8574-e6752ab22716" (UID: "22f220aa-6950-47d0-8574-e6752ab22716"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:16:41.734600 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.734536 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "22f220aa-6950-47d0-8574-e6752ab22716" (UID: "22f220aa-6950-47d0-8574-e6752ab22716"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:16:41.734654 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.734642 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-workload-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:16:41.734715 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.734660 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-credential-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:16:41.735099 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.734845 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f220aa-6950-47d0-8574-e6752ab22716-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "22f220aa-6950-47d0-8574-e6752ab22716" (UID: "22f220aa-6950-47d0-8574-e6752ab22716"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:16:41.735099 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.734944 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "22f220aa-6950-47d0-8574-e6752ab22716" (UID: "22f220aa-6950-47d0-8574-e6752ab22716"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:16:41.735290 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.735272 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-istio-data" (OuterVolumeSpecName: "istio-data") pod "22f220aa-6950-47d0-8574-e6752ab22716" (UID: "22f220aa-6950-47d0-8574-e6752ab22716"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:16:41.736750 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.736724 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "22f220aa-6950-47d0-8574-e6752ab22716" (UID: "22f220aa-6950-47d0-8574-e6752ab22716"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:16:41.736853 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.736788 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22f220aa-6950-47d0-8574-e6752ab22716-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "22f220aa-6950-47d0-8574-e6752ab22716" (UID: "22f220aa-6950-47d0-8574-e6752ab22716"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 22:16:41.736853 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.736815 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f220aa-6950-47d0-8574-e6752ab22716-kube-api-access-djm8d" (OuterVolumeSpecName: "kube-api-access-djm8d") pod "22f220aa-6950-47d0-8574-e6752ab22716" (UID: "22f220aa-6950-47d0-8574-e6752ab22716"). InnerVolumeSpecName "kube-api-access-djm8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:16:41.737011 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.736997 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f220aa-6950-47d0-8574-e6752ab22716-istio-token" (OuterVolumeSpecName: "istio-token") pod "22f220aa-6950-47d0-8574-e6752ab22716" (UID: "22f220aa-6950-47d0-8574-e6752ab22716"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:16:41.836669 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.836630 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f220aa-6950-47d0-8574-e6752ab22716-istio-token\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:16:41.836669 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.836662 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f220aa-6950-47d0-8574-e6752ab22716-istio-podinfo\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:16:41.836669 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.836671 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-istio-envoy\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:16:41.836923 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.836680 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djm8d\" (UniqueName: \"kubernetes.io/projected/22f220aa-6950-47d0-8574-e6752ab22716-kube-api-access-djm8d\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:16:41.836923 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.836690 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-workload-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:16:41.836923 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.836700 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f220aa-6950-47d0-8574-e6752ab22716-istiod-ca-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:16:41.836923 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:41.836710 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f220aa-6950-47d0-8574-e6752ab22716-istio-data\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:16:42.459275 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:42.459234 2580 generic.go:356] "Generic (PLEG): container finished" podID="22f220aa-6950-47d0-8574-e6752ab22716" containerID="6b7bc9ae178e83c95a37227dfdfc7a9aba1747a2fe662162c238cd1318242753" exitCode=0 Jul 21 22:16:42.459275 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:42.459271 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" event={"ID":"22f220aa-6950-47d0-8574-e6752ab22716","Type":"ContainerDied","Data":"6b7bc9ae178e83c95a37227dfdfc7a9aba1747a2fe662162c238cd1318242753"} Jul 21 22:16:42.459578 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:42.459283 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" Jul 21 22:16:42.459578 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:42.459327 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm" event={"ID":"22f220aa-6950-47d0-8574-e6752ab22716","Type":"ContainerDied","Data":"660e46fb2465feed832753163df715e931b97940eb56fd617433b5c30f04d0ea"} Jul 21 22:16:42.459578 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:42.459359 2580 scope.go:117] "RemoveContainer" containerID="6b7bc9ae178e83c95a37227dfdfc7a9aba1747a2fe662162c238cd1318242753" Jul 21 22:16:42.469198 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:42.469176 2580 scope.go:117] "RemoveContainer" containerID="7989791097e274036896424638bdd0f88f005f7d79299843780b7152a0a698c7" Jul 21 22:16:42.477415 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:42.477400 2580 scope.go:117] "RemoveContainer" containerID="6b7bc9ae178e83c95a37227dfdfc7a9aba1747a2fe662162c238cd1318242753" Jul 21 22:16:42.477709 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:16:42.477686 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b7bc9ae178e83c95a37227dfdfc7a9aba1747a2fe662162c238cd1318242753\": container with ID starting with 6b7bc9ae178e83c95a37227dfdfc7a9aba1747a2fe662162c238cd1318242753 not found: ID does not exist" containerID="6b7bc9ae178e83c95a37227dfdfc7a9aba1747a2fe662162c238cd1318242753" Jul 21 22:16:42.477813 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:42.477716 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b7bc9ae178e83c95a37227dfdfc7a9aba1747a2fe662162c238cd1318242753"} err="failed to get container status \"6b7bc9ae178e83c95a37227dfdfc7a9aba1747a2fe662162c238cd1318242753\": rpc error: code = NotFound desc = could not find container \"6b7bc9ae178e83c95a37227dfdfc7a9aba1747a2fe662162c238cd1318242753\": container with ID starting with 6b7bc9ae178e83c95a37227dfdfc7a9aba1747a2fe662162c238cd1318242753 not found: ID does not exist" Jul 21 22:16:42.477813 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:42.477734 2580 scope.go:117] "RemoveContainer" containerID="7989791097e274036896424638bdd0f88f005f7d79299843780b7152a0a698c7" Jul 21 22:16:42.478019 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:16:42.477994 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7989791097e274036896424638bdd0f88f005f7d79299843780b7152a0a698c7\": container with ID starting with 7989791097e274036896424638bdd0f88f005f7d79299843780b7152a0a698c7 not found: ID does not exist" containerID="7989791097e274036896424638bdd0f88f005f7d79299843780b7152a0a698c7" Jul 21 22:16:42.478069 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:42.478025 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7989791097e274036896424638bdd0f88f005f7d79299843780b7152a0a698c7"} err="failed to get container status \"7989791097e274036896424638bdd0f88f005f7d79299843780b7152a0a698c7\": rpc error: code = NotFound desc = could not find container \"7989791097e274036896424638bdd0f88f005f7d79299843780b7152a0a698c7\": container with ID starting with 7989791097e274036896424638bdd0f88f005f7d79299843780b7152a0a698c7 not found: ID does not exist" Jul 21 22:16:42.485185 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:42.485164 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm"] Jul 21 22:16:42.493361 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:42.493342 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-678339c2-openshift-default-7684f8df7-c7mfm"] Jul 21 22:16:42.780708 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:42.780678 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22f220aa-6950-47d0-8574-e6752ab22716" path="/var/lib/kubelet/pods/22f220aa-6950-47d0-8574-e6752ab22716/volumes" Jul 21 22:16:54.533383 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:54.533338 2580 generic.go:356] "Generic (PLEG): container finished" podID="41bbcc55-4322-4c00-94d2-2184fad4d62c" containerID="7f4b2ccc6a5cfd9bf60637be8bf76aeb61b362b51bebc424f69e27556771f59a" exitCode=0 Jul 21 22:16:54.533810 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:54.533417 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" event={"ID":"41bbcc55-4322-4c00-94d2-2184fad4d62c","Type":"ContainerDied","Data":"7f4b2ccc6a5cfd9bf60637be8bf76aeb61b362b51bebc424f69e27556771f59a"} Jul 21 22:16:54.533855 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:54.533830 2580 scope.go:117] "RemoveContainer" containerID="7f4b2ccc6a5cfd9bf60637be8bf76aeb61b362b51bebc424f69e27556771f59a" Jul 21 22:16:54.534900 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:54.534872 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:16:54.534971 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:54.534954 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:16:54.535015 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:54.535002 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:16:55.448913 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:55.448874 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:55.542321 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:55.542281 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" event={"ID":"41bbcc55-4322-4c00-94d2-2184fad4d62c","Type":"ContainerStarted","Data":"866c544973740db3ce71a8b4f2aef69840a4497765bc801a52ced6ff5ac78bb9"} Jul 21 22:16:55.829807 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:55.829764 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:55.830849 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:55.830824 2580 patch_prober.go:28] interesting pod/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 21 22:16:55.830968 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:55.830872 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" podUID="41bbcc55-4322-4c00-94d2-2184fad4d62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 21 22:16:56.829489 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:56.829454 2580 patch_prober.go:28] interesting pod/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 21 22:16:56.829890 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:56.829505 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" podUID="41bbcc55-4322-4c00-94d2-2184fad4d62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 21 22:16:57.830283 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:57.830247 2580 patch_prober.go:28] interesting pod/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 21 22:16:57.830703 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:57.830307 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" podUID="41bbcc55-4322-4c00-94d2-2184fad4d62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 21 22:16:58.829561 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:58.829519 2580 patch_prober.go:28] interesting pod/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 21 22:16:58.829759 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:58.829583 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" podUID="41bbcc55-4322-4c00-94d2-2184fad4d62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 21 22:16:59.832938 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:59.832909 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:59.833335 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:59.833183 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:16:59.833898 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:16:59.833880 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:17:20.186572 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:20.186539 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2"] Jul 21 22:17:20.187104 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:20.186811 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" podUID="41bbcc55-4322-4c00-94d2-2184fad4d62c" containerName="istio-proxy" containerID="cri-o://866c544973740db3ce71a8b4f2aef69840a4497765bc801a52ced6ff5ac78bb9" gracePeriod=30 Jul 21 22:17:20.981401 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:20.981341 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d"] Jul 21 22:17:20.982080 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:20.982057 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f220aa-6950-47d0-8574-e6752ab22716" containerName="istio-proxy" Jul 21 22:17:20.982080 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:20.982081 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f220aa-6950-47d0-8574-e6752ab22716" containerName="istio-proxy" Jul 21 22:17:20.982219 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:20.982098 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f220aa-6950-47d0-8574-e6752ab22716" containerName="istio-proxy" Jul 21 22:17:20.982219 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:20.982107 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f220aa-6950-47d0-8574-e6752ab22716" containerName="istio-proxy" Jul 21 22:17:20.982317 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:20.982232 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="22f220aa-6950-47d0-8574-e6752ab22716" containerName="istio-proxy" Jul 21 22:17:20.982317 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:20.982244 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="22f220aa-6950-47d0-8574-e6752ab22716" containerName="istio-proxy" Jul 21 22:17:20.985742 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:20.985724 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:20.989275 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:20.989257 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bc2a2d3f-openshift-default-dockercfg-9s56f\"" Jul 21 22:17:20.998218 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:20.998198 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d"] Jul 21 22:17:21.130683 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.130650 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-istio-envoy\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.130683 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.130685 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-workload-certs\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.130921 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.130715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8701da77-52b8-489b-9331-df57ecdac65e-istio-token\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.130921 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.130816 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dddrn\" (UniqueName: \"kubernetes.io/projected/8701da77-52b8-489b-9331-df57ecdac65e-kube-api-access-dddrn\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.130921 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.130917 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-istio-data\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.131054 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.130939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8701da77-52b8-489b-9331-df57ecdac65e-istiod-ca-cert\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.131054 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.130958 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-credential-socket\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.131054 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.130979 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8701da77-52b8-489b-9331-df57ecdac65e-istio-podinfo\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.131054 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.131051 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-workload-socket\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.232017 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.231926 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddrn\" (UniqueName: \"kubernetes.io/projected/8701da77-52b8-489b-9331-df57ecdac65e-kube-api-access-dddrn\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.232480 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.232052 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-istio-data\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.232480 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.232085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8701da77-52b8-489b-9331-df57ecdac65e-istiod-ca-cert\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.232480 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.232114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-credential-socket\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.232480 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.232139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8701da77-52b8-489b-9331-df57ecdac65e-istio-podinfo\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.232480 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.232195 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-workload-socket\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.232480 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.232243 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-istio-envoy\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.232480 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.232272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-workload-certs\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.232480 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.232312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8701da77-52b8-489b-9331-df57ecdac65e-istio-token\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.232996 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.232915 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8701da77-52b8-489b-9331-df57ecdac65e-istiod-ca-cert\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.232996 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.232920 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-workload-socket\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.232996 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.232959 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-credential-socket\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.233413 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.233392 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-workload-certs\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.233672 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.233651 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-istio-data\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.235357 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.235326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8701da77-52b8-489b-9331-df57ecdac65e-istio-podinfo\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.235652 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.235633 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-istio-envoy\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.242223 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.242205 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dddrn\" (UniqueName: \"kubernetes.io/projected/8701da77-52b8-489b-9331-df57ecdac65e-kube-api-access-dddrn\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.245703 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.245686 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8701da77-52b8-489b-9331-df57ecdac65e-istio-token\") pod \"e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.300233 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.300213 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:21.441498 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.441475 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d"] Jul 21 22:17:21.443308 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:17:21.443276 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8701da77_52b8_489b_9331_df57ecdac65e.slice/crio-2f0a831676a076f444c185f29acd084464f57216ec1e8423d817aeac3bb8a610 WatchSource:0}: Error finding container 2f0a831676a076f444c185f29acd084464f57216ec1e8423d817aeac3bb8a610: Status 404 returned error can't find the container with id 2f0a831676a076f444c185f29acd084464f57216ec1e8423d817aeac3bb8a610 Jul 21 22:17:21.445554 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.445518 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:17:21.445640 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.445587 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:17:21.445640 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.445615 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:17:21.678537 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.678502 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" event={"ID":"8701da77-52b8-489b-9331-df57ecdac65e","Type":"ContainerStarted","Data":"a67d9f998596b0c7f09093bf4554c2ad15b2b4931908f8cdd5c0955dcada6fe0"} Jul 21 22:17:21.678710 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.678565 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" event={"ID":"8701da77-52b8-489b-9331-df57ecdac65e","Type":"ContainerStarted","Data":"2f0a831676a076f444c185f29acd084464f57216ec1e8423d817aeac3bb8a610"} Jul 21 22:17:21.699904 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:21.699838 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" podStartSLOduration=1.699820103 podStartE2EDuration="1.699820103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:17:21.699097005 +0000 UTC m=+1719.462331293" watchObservedRunningTime="2026-07-21 22:17:21.699820103 +0000 UTC m=+1719.463054365" Jul 21 22:17:22.301059 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:22.301034 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:22.306021 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:22.305998 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:22.683551 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:22.683476 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:22.684431 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:22.684407 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:25.445084 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.445061 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:17:25.580574 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.580535 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-credential-socket\") pod \"41bbcc55-4322-4c00-94d2-2184fad4d62c\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " Jul 21 22:17:25.580749 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.580592 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-envoy\") pod \"41bbcc55-4322-4c00-94d2-2184fad4d62c\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " Jul 21 22:17:25.580749 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.580626 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-workload-socket\") pod \"41bbcc55-4322-4c00-94d2-2184fad4d62c\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " Jul 21 22:17:25.580749 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.580645 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-workload-certs\") pod \"41bbcc55-4322-4c00-94d2-2184fad4d62c\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " Jul 21 22:17:25.580749 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.580665 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-podinfo\") pod \"41bbcc55-4322-4c00-94d2-2184fad4d62c\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " Jul 21 22:17:25.580749 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.580720 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41bbcc55-4322-4c00-94d2-2184fad4d62c-istiod-ca-cert\") pod \"41bbcc55-4322-4c00-94d2-2184fad4d62c\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " Jul 21 22:17:25.581000 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.580784 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-data\") pod \"41bbcc55-4322-4c00-94d2-2184fad4d62c\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " Jul 21 22:17:25.581000 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.580810 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "41bbcc55-4322-4c00-94d2-2184fad4d62c" (UID: "41bbcc55-4322-4c00-94d2-2184fad4d62c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:17:25.581000 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.580831 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-token\") pod \"41bbcc55-4322-4c00-94d2-2184fad4d62c\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " Jul 21 22:17:25.581000 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.580862 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qrx4\" (UniqueName: \"kubernetes.io/projected/41bbcc55-4322-4c00-94d2-2184fad4d62c-kube-api-access-8qrx4\") pod \"41bbcc55-4322-4c00-94d2-2184fad4d62c\" (UID: \"41bbcc55-4322-4c00-94d2-2184fad4d62c\") " Jul 21 22:17:25.581000 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.580876 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "41bbcc55-4322-4c00-94d2-2184fad4d62c" (UID: "41bbcc55-4322-4c00-94d2-2184fad4d62c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:17:25.581262 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.581186 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41bbcc55-4322-4c00-94d2-2184fad4d62c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "41bbcc55-4322-4c00-94d2-2184fad4d62c" (UID: "41bbcc55-4322-4c00-94d2-2184fad4d62c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:17:25.581262 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.581203 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-credential-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:17:25.581262 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.581223 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-workload-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:17:25.581262 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.581252 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "41bbcc55-4322-4c00-94d2-2184fad4d62c" (UID: "41bbcc55-4322-4c00-94d2-2184fad4d62c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:17:25.581816 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.581791 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-data" (OuterVolumeSpecName: "istio-data") pod "41bbcc55-4322-4c00-94d2-2184fad4d62c" (UID: "41bbcc55-4322-4c00-94d2-2184fad4d62c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:17:25.583140 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.583111 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41bbcc55-4322-4c00-94d2-2184fad4d62c-kube-api-access-8qrx4" (OuterVolumeSpecName: "kube-api-access-8qrx4") pod "41bbcc55-4322-4c00-94d2-2184fad4d62c" (UID: "41bbcc55-4322-4c00-94d2-2184fad4d62c"). InnerVolumeSpecName "kube-api-access-8qrx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:17:25.583221 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.583131 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-token" (OuterVolumeSpecName: "istio-token") pod "41bbcc55-4322-4c00-94d2-2184fad4d62c" (UID: "41bbcc55-4322-4c00-94d2-2184fad4d62c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:17:25.583221 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.583157 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "41bbcc55-4322-4c00-94d2-2184fad4d62c" (UID: "41bbcc55-4322-4c00-94d2-2184fad4d62c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:17:25.583221 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.583159 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "41bbcc55-4322-4c00-94d2-2184fad4d62c" (UID: "41bbcc55-4322-4c00-94d2-2184fad4d62c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 22:17:25.682729 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.682687 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-envoy\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:17:25.682729 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.682717 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-workload-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:17:25.682729 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.682730 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-podinfo\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:17:25.682917 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.682741 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41bbcc55-4322-4c00-94d2-2184fad4d62c-istiod-ca-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:17:25.682917 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.682755 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-data\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:17:25.682917 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.682765 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41bbcc55-4322-4c00-94d2-2184fad4d62c-istio-token\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:17:25.682917 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.682775 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qrx4\" (UniqueName: \"kubernetes.io/projected/41bbcc55-4322-4c00-94d2-2184fad4d62c-kube-api-access-8qrx4\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:17:25.702776 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.702745 2580 generic.go:356] "Generic (PLEG): container finished" podID="41bbcc55-4322-4c00-94d2-2184fad4d62c" containerID="866c544973740db3ce71a8b4f2aef69840a4497765bc801a52ced6ff5ac78bb9" exitCode=0 Jul 21 22:17:25.702865 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.702805 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" Jul 21 22:17:25.702865 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.702833 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" event={"ID":"41bbcc55-4322-4c00-94d2-2184fad4d62c","Type":"ContainerDied","Data":"866c544973740db3ce71a8b4f2aef69840a4497765bc801a52ced6ff5ac78bb9"} Jul 21 22:17:25.702944 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.702896 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2" event={"ID":"41bbcc55-4322-4c00-94d2-2184fad4d62c","Type":"ContainerDied","Data":"e2901cf14e493d59684d050708c7209adec4eacab2a0679432366783df245690"} Jul 21 22:17:25.702944 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.702926 2580 scope.go:117] "RemoveContainer" containerID="866c544973740db3ce71a8b4f2aef69840a4497765bc801a52ced6ff5ac78bb9" Jul 21 22:17:25.712472 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.712456 2580 scope.go:117] "RemoveContainer" containerID="7f4b2ccc6a5cfd9bf60637be8bf76aeb61b362b51bebc424f69e27556771f59a" Jul 21 22:17:25.720803 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.720781 2580 scope.go:117] "RemoveContainer" containerID="866c544973740db3ce71a8b4f2aef69840a4497765bc801a52ced6ff5ac78bb9" Jul 21 22:17:25.721094 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:17:25.721075 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"866c544973740db3ce71a8b4f2aef69840a4497765bc801a52ced6ff5ac78bb9\": container with ID starting with 866c544973740db3ce71a8b4f2aef69840a4497765bc801a52ced6ff5ac78bb9 not found: ID does not exist" containerID="866c544973740db3ce71a8b4f2aef69840a4497765bc801a52ced6ff5ac78bb9" Jul 21 22:17:25.721139 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.721102 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"866c544973740db3ce71a8b4f2aef69840a4497765bc801a52ced6ff5ac78bb9"} err="failed to get container status \"866c544973740db3ce71a8b4f2aef69840a4497765bc801a52ced6ff5ac78bb9\": rpc error: code = NotFound desc = could not find container \"866c544973740db3ce71a8b4f2aef69840a4497765bc801a52ced6ff5ac78bb9\": container with ID starting with 866c544973740db3ce71a8b4f2aef69840a4497765bc801a52ced6ff5ac78bb9 not found: ID does not exist" Jul 21 22:17:25.721139 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.721118 2580 scope.go:117] "RemoveContainer" containerID="7f4b2ccc6a5cfd9bf60637be8bf76aeb61b362b51bebc424f69e27556771f59a" Jul 21 22:17:25.721368 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:17:25.721351 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f4b2ccc6a5cfd9bf60637be8bf76aeb61b362b51bebc424f69e27556771f59a\": container with ID starting with 7f4b2ccc6a5cfd9bf60637be8bf76aeb61b362b51bebc424f69e27556771f59a not found: ID does not exist" containerID="7f4b2ccc6a5cfd9bf60637be8bf76aeb61b362b51bebc424f69e27556771f59a" Jul 21 22:17:25.721448 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.721402 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f4b2ccc6a5cfd9bf60637be8bf76aeb61b362b51bebc424f69e27556771f59a"} err="failed to get container status \"7f4b2ccc6a5cfd9bf60637be8bf76aeb61b362b51bebc424f69e27556771f59a\": rpc error: code = NotFound desc = could not find container \"7f4b2ccc6a5cfd9bf60637be8bf76aeb61b362b51bebc424f69e27556771f59a\": container with ID starting with 7f4b2ccc6a5cfd9bf60637be8bf76aeb61b362b51bebc424f69e27556771f59a not found: ID does not exist" Jul 21 22:17:25.725207 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.725186 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2"] Jul 21 22:17:25.731686 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:25.731667 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-dd422c5b-gw-openshift-default-fb64bc6ff-x7xw2"] Jul 21 22:17:26.779539 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:26.779507 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41bbcc55-4322-4c00-94d2-2184fad4d62c" path="/var/lib/kubelet/pods/41bbcc55-4322-4c00-94d2-2184fad4d62c/volumes" Jul 21 22:17:37.685434 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:37.685397 2580 patch_prober.go:28] interesting pod/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 21 22:17:37.685840 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:37.685453 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" podUID="8701da77-52b8-489b-9331-df57ecdac65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 21 22:17:37.765907 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:37.765877 2580 generic.go:356] "Generic (PLEG): container finished" podID="8701da77-52b8-489b-9331-df57ecdac65e" containerID="a67d9f998596b0c7f09093bf4554c2ad15b2b4931908f8cdd5c0955dcada6fe0" exitCode=0 Jul 21 22:17:37.766079 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:37.765959 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" event={"ID":"8701da77-52b8-489b-9331-df57ecdac65e","Type":"ContainerDied","Data":"a67d9f998596b0c7f09093bf4554c2ad15b2b4931908f8cdd5c0955dcada6fe0"} Jul 21 22:17:37.766460 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:37.766446 2580 scope.go:117] "RemoveContainer" containerID="a67d9f998596b0c7f09093bf4554c2ad15b2b4931908f8cdd5c0955dcada6fe0" Jul 21 22:17:37.767439 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:37.767407 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:17:37.767572 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:37.767475 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:17:37.767572 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:37.767515 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:17:38.786640 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:38.786596 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" event={"ID":"8701da77-52b8-489b-9331-df57ecdac65e","Type":"ContainerStarted","Data":"2e105747fdd0e4171c28a1916c9f301d5ea49b908d171b77f4a4dec19df572ca"} Jul 21 22:17:39.301286 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:39.301246 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:39.301552 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:39.301530 2580 patch_prober.go:28] interesting pod/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 21 22:17:39.301622 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:39.301601 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" podUID="8701da77-52b8-489b-9331-df57ecdac65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 21 22:17:40.301099 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:40.301059 2580 patch_prober.go:28] interesting pod/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 21 22:17:40.301533 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:40.301139 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" podUID="8701da77-52b8-489b-9331-df57ecdac65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 21 22:17:41.301420 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:41.301386 2580 patch_prober.go:28] interesting pod/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 21 22:17:41.301785 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:41.301445 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" podUID="8701da77-52b8-489b-9331-df57ecdac65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 21 22:17:42.301259 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:42.301225 2580 patch_prober.go:28] interesting pod/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 21 22:17:42.301468 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:42.301281 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" podUID="8701da77-52b8-489b-9331-df57ecdac65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 21 22:17:43.305107 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:43.305079 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:43.305481 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:43.305412 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:17:43.305991 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:17:43.305975 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:18:05.209919 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:05.209881 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d"] Jul 21 22:18:05.210388 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:05.210127 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" podUID="8701da77-52b8-489b-9331-df57ecdac65e" containerName="istio-proxy" containerID="cri-o://2e105747fdd0e4171c28a1916c9f301d5ea49b908d171b77f4a4dec19df572ca" gracePeriod=30 Jul 21 22:18:05.948332 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:05.948300 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm"] Jul 21 22:18:05.948828 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:05.948810 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41bbcc55-4322-4c00-94d2-2184fad4d62c" containerName="istio-proxy" Jul 21 22:18:05.948923 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:05.948830 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="41bbcc55-4322-4c00-94d2-2184fad4d62c" containerName="istio-proxy" Jul 21 22:18:05.948923 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:05.948848 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41bbcc55-4322-4c00-94d2-2184fad4d62c" containerName="istio-proxy" Jul 21 22:18:05.948923 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:05.948856 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="41bbcc55-4322-4c00-94d2-2184fad4d62c" containerName="istio-proxy" Jul 21 22:18:05.949081 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:05.948980 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="41bbcc55-4322-4c00-94d2-2184fad4d62c" containerName="istio-proxy" Jul 21 22:18:05.949081 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:05.948993 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="41bbcc55-4322-4c00-94d2-2184fad4d62c" containerName="istio-proxy" Jul 21 22:18:05.953506 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:05.953484 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:05.955966 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:05.955950 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-85902df0-openshift-default-dockercfg-5qmgn\"" Jul 21 22:18:05.963314 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:05.963293 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm"] Jul 21 22:18:06.080817 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.080782 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-workload-certs\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.080817 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.080829 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-credential-socket\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.081055 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.080855 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-envoy\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.081055 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.080944 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-podinfo\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.081055 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.080983 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shfg6\" (UniqueName: \"kubernetes.io/projected/9c9eee7f-ee88-4399-8684-1e83baa3e25a-kube-api-access-shfg6\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.081055 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.081008 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-workload-socket\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.081252 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.081117 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istiod-ca-cert\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.081252 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.081162 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-data\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.081252 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.081213 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-token\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.181858 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.181826 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istiod-ca-cert\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.182049 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.181868 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-data\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.182049 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.181994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-token\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.182049 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.182043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-workload-certs\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.182235 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.182078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-credential-socket\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.182235 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.182108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-envoy\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.182235 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.182173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-podinfo\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.182235 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.182211 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shfg6\" (UniqueName: \"kubernetes.io/projected/9c9eee7f-ee88-4399-8684-1e83baa3e25a-kube-api-access-shfg6\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.182471 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.182243 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-workload-socket\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.182471 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.182260 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-data\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.182578 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.182512 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-workload-certs\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.182578 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.182534 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-credential-socket\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.182578 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.182565 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-workload-socket\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.182691 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.182672 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istiod-ca-cert\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.184652 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.184623 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-envoy\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.184835 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.184816 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-podinfo\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.194403 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.194358 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-token\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.194504 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.194488 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shfg6\" (UniqueName: \"kubernetes.io/projected/9c9eee7f-ee88-4399-8684-1e83baa3e25a-kube-api-access-shfg6\") pod \"e2e-derive-85902df0-openshift-default-7c87597755-592bm\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.265472 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.265449 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:06.413948 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.413879 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm"] Jul 21 22:18:06.416241 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:18:06.416219 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9eee7f_ee88_4399_8684_1e83baa3e25a.slice/crio-6d95db93ca7339a26578329ce32a6595a93f7f05c50fdf97f49c3f0846cff2ad WatchSource:0}: Error finding container 6d95db93ca7339a26578329ce32a6595a93f7f05c50fdf97f49c3f0846cff2ad: Status 404 returned error can't find the container with id 6d95db93ca7339a26578329ce32a6595a93f7f05c50fdf97f49c3f0846cff2ad Jul 21 22:18:06.418366 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.418324 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:18:06.418476 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.418420 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:18:06.418476 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.418452 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:18:06.934482 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.934448 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" event={"ID":"9c9eee7f-ee88-4399-8684-1e83baa3e25a","Type":"ContainerStarted","Data":"41513cdc74313c2498a3d4c8629da4c40fdf60ad429e72608046dec3f9923024"} Jul 21 22:18:06.934655 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.934496 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" event={"ID":"9c9eee7f-ee88-4399-8684-1e83baa3e25a","Type":"ContainerStarted","Data":"6d95db93ca7339a26578329ce32a6595a93f7f05c50fdf97f49c3f0846cff2ad"} Jul 21 22:18:06.957844 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:06.957801 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" podStartSLOduration=1.9577865060000001 podStartE2EDuration="1.957786506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:18:06.95523785 +0000 UTC m=+1764.718472111" watchObservedRunningTime="2026-07-21 22:18:06.957786506 +0000 UTC m=+1764.721020766" Jul 21 22:18:07.266562 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:07.266534 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:07.268318 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:07.268289 2580 patch_prober.go:28] interesting pod/e2e-derive-85902df0-openshift-default-7c87597755-592bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 21 22:18:07.268467 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:07.268339 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" podUID="9c9eee7f-ee88-4399-8684-1e83baa3e25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 21 22:18:08.269696 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:08.269666 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:08.945383 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:08.945350 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:08.946269 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:08.946245 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:10.468161 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.468135 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:18:10.524875 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.524845 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-istio-envoy\") pod \"8701da77-52b8-489b-9331-df57ecdac65e\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " Jul 21 22:18:10.525023 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.524903 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-credential-socket\") pod \"8701da77-52b8-489b-9331-df57ecdac65e\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " Jul 21 22:18:10.525023 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.524928 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-workload-socket\") pod \"8701da77-52b8-489b-9331-df57ecdac65e\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " Jul 21 22:18:10.525023 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.524956 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8701da77-52b8-489b-9331-df57ecdac65e-istiod-ca-cert\") pod \"8701da77-52b8-489b-9331-df57ecdac65e\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " Jul 21 22:18:10.525023 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.524977 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8701da77-52b8-489b-9331-df57ecdac65e-istio-podinfo\") pod \"8701da77-52b8-489b-9331-df57ecdac65e\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " Jul 21 22:18:10.525195 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.525052 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-workload-certs\") pod \"8701da77-52b8-489b-9331-df57ecdac65e\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " Jul 21 22:18:10.525195 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.525110 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-istio-data\") pod \"8701da77-52b8-489b-9331-df57ecdac65e\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " Jul 21 22:18:10.525195 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.525138 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dddrn\" (UniqueName: \"kubernetes.io/projected/8701da77-52b8-489b-9331-df57ecdac65e-kube-api-access-dddrn\") pod \"8701da77-52b8-489b-9331-df57ecdac65e\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " Jul 21 22:18:10.525346 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.525199 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8701da77-52b8-489b-9331-df57ecdac65e-istio-token\") pod \"8701da77-52b8-489b-9331-df57ecdac65e\" (UID: \"8701da77-52b8-489b-9331-df57ecdac65e\") " Jul 21 22:18:10.525346 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.525212 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8701da77-52b8-489b-9331-df57ecdac65e" (UID: "8701da77-52b8-489b-9331-df57ecdac65e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:18:10.525346 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.525252 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8701da77-52b8-489b-9331-df57ecdac65e" (UID: "8701da77-52b8-489b-9331-df57ecdac65e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:18:10.525521 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.525406 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8701da77-52b8-489b-9331-df57ecdac65e" (UID: "8701da77-52b8-489b-9331-df57ecdac65e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:18:10.525521 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.525478 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8701da77-52b8-489b-9331-df57ecdac65e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8701da77-52b8-489b-9331-df57ecdac65e" (UID: "8701da77-52b8-489b-9331-df57ecdac65e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:18:10.525640 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.525601 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-credential-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:18:10.525640 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.525619 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-workload-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:18:10.525640 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.525635 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8701da77-52b8-489b-9331-df57ecdac65e-istiod-ca-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:18:10.525788 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.525647 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-workload-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:18:10.526163 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.526137 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-istio-data" (OuterVolumeSpecName: "istio-data") pod "8701da77-52b8-489b-9331-df57ecdac65e" (UID: "8701da77-52b8-489b-9331-df57ecdac65e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:18:10.527438 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.527411 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8701da77-52b8-489b-9331-df57ecdac65e-kube-api-access-dddrn" (OuterVolumeSpecName: "kube-api-access-dddrn") pod "8701da77-52b8-489b-9331-df57ecdac65e" (UID: "8701da77-52b8-489b-9331-df57ecdac65e"). InnerVolumeSpecName "kube-api-access-dddrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:18:10.527545 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.527432 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8701da77-52b8-489b-9331-df57ecdac65e-istio-token" (OuterVolumeSpecName: "istio-token") pod "8701da77-52b8-489b-9331-df57ecdac65e" (UID: "8701da77-52b8-489b-9331-df57ecdac65e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:18:10.527682 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.527660 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8701da77-52b8-489b-9331-df57ecdac65e" (UID: "8701da77-52b8-489b-9331-df57ecdac65e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:18:10.527726 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.527674 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8701da77-52b8-489b-9331-df57ecdac65e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8701da77-52b8-489b-9331-df57ecdac65e" (UID: "8701da77-52b8-489b-9331-df57ecdac65e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 22:18:10.626555 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.626532 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-istio-data\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:18:10.626555 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.626553 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dddrn\" (UniqueName: \"kubernetes.io/projected/8701da77-52b8-489b-9331-df57ecdac65e-kube-api-access-dddrn\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:18:10.626675 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.626565 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8701da77-52b8-489b-9331-df57ecdac65e-istio-token\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:18:10.626675 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.626573 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8701da77-52b8-489b-9331-df57ecdac65e-istio-envoy\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:18:10.626675 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.626580 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8701da77-52b8-489b-9331-df57ecdac65e-istio-podinfo\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:18:10.957631 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.957599 2580 generic.go:356] "Generic (PLEG): container finished" podID="8701da77-52b8-489b-9331-df57ecdac65e" containerID="2e105747fdd0e4171c28a1916c9f301d5ea49b908d171b77f4a4dec19df572ca" exitCode=0 Jul 21 22:18:10.957770 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.957665 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" Jul 21 22:18:10.957770 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.957687 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" event={"ID":"8701da77-52b8-489b-9331-df57ecdac65e","Type":"ContainerDied","Data":"2e105747fdd0e4171c28a1916c9f301d5ea49b908d171b77f4a4dec19df572ca"} Jul 21 22:18:10.957770 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.957747 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d" event={"ID":"8701da77-52b8-489b-9331-df57ecdac65e","Type":"ContainerDied","Data":"2f0a831676a076f444c185f29acd084464f57216ec1e8423d817aeac3bb8a610"} Jul 21 22:18:10.957878 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.957777 2580 scope.go:117] "RemoveContainer" containerID="2e105747fdd0e4171c28a1916c9f301d5ea49b908d171b77f4a4dec19df572ca" Jul 21 22:18:10.966737 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.966719 2580 scope.go:117] "RemoveContainer" containerID="a67d9f998596b0c7f09093bf4554c2ad15b2b4931908f8cdd5c0955dcada6fe0" Jul 21 22:18:10.976691 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.976672 2580 scope.go:117] "RemoveContainer" containerID="2e105747fdd0e4171c28a1916c9f301d5ea49b908d171b77f4a4dec19df572ca" Jul 21 22:18:10.976989 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:18:10.976964 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e105747fdd0e4171c28a1916c9f301d5ea49b908d171b77f4a4dec19df572ca\": container with ID starting with 2e105747fdd0e4171c28a1916c9f301d5ea49b908d171b77f4a4dec19df572ca not found: ID does not exist" containerID="2e105747fdd0e4171c28a1916c9f301d5ea49b908d171b77f4a4dec19df572ca" Jul 21 22:18:10.977047 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.977004 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e105747fdd0e4171c28a1916c9f301d5ea49b908d171b77f4a4dec19df572ca"} err="failed to get container status \"2e105747fdd0e4171c28a1916c9f301d5ea49b908d171b77f4a4dec19df572ca\": rpc error: code = NotFound desc = could not find container \"2e105747fdd0e4171c28a1916c9f301d5ea49b908d171b77f4a4dec19df572ca\": container with ID starting with 2e105747fdd0e4171c28a1916c9f301d5ea49b908d171b77f4a4dec19df572ca not found: ID does not exist" Jul 21 22:18:10.977047 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.977028 2580 scope.go:117] "RemoveContainer" containerID="a67d9f998596b0c7f09093bf4554c2ad15b2b4931908f8cdd5c0955dcada6fe0" Jul 21 22:18:10.977306 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:18:10.977283 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a67d9f998596b0c7f09093bf4554c2ad15b2b4931908f8cdd5c0955dcada6fe0\": container with ID starting with a67d9f998596b0c7f09093bf4554c2ad15b2b4931908f8cdd5c0955dcada6fe0 not found: ID does not exist" containerID="a67d9f998596b0c7f09093bf4554c2ad15b2b4931908f8cdd5c0955dcada6fe0" Jul 21 22:18:10.977394 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.977315 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a67d9f998596b0c7f09093bf4554c2ad15b2b4931908f8cdd5c0955dcada6fe0"} err="failed to get container status \"a67d9f998596b0c7f09093bf4554c2ad15b2b4931908f8cdd5c0955dcada6fe0\": rpc error: code = NotFound desc = could not find container \"a67d9f998596b0c7f09093bf4554c2ad15b2b4931908f8cdd5c0955dcada6fe0\": container with ID starting with a67d9f998596b0c7f09093bf4554c2ad15b2b4931908f8cdd5c0955dcada6fe0 not found: ID does not exist" Jul 21 22:18:10.978135 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.978118 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d"] Jul 21 22:18:10.982932 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:10.982888 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bc2a2d3f-openshift-default-58688554c7-7pf8d"] Jul 21 22:18:12.809290 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:12.809254 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8701da77-52b8-489b-9331-df57ecdac65e" path="/var/lib/kubelet/pods/8701da77-52b8-489b-9331-df57ecdac65e/volumes" Jul 21 22:18:20.022087 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:20.022053 2580 generic.go:356] "Generic (PLEG): container finished" podID="9c9eee7f-ee88-4399-8684-1e83baa3e25a" containerID="41513cdc74313c2498a3d4c8629da4c40fdf60ad429e72608046dec3f9923024" exitCode=0 Jul 21 22:18:20.022565 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:20.022132 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" event={"ID":"9c9eee7f-ee88-4399-8684-1e83baa3e25a","Type":"ContainerDied","Data":"41513cdc74313c2498a3d4c8629da4c40fdf60ad429e72608046dec3f9923024"} Jul 21 22:18:20.022565 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:20.022555 2580 scope.go:117] "RemoveContainer" containerID="41513cdc74313c2498a3d4c8629da4c40fdf60ad429e72608046dec3f9923024" Jul 21 22:18:20.023631 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:20.023603 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:18:20.023695 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:20.023673 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:18:20.023739 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:20.023704 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:18:21.029986 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:21.029954 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" event={"ID":"9c9eee7f-ee88-4399-8684-1e83baa3e25a","Type":"ContainerStarted","Data":"8cd8a02b66754baa424fc83faf417a00ed39d8c30d0b37702d1e4ea607415f5f"} Jul 21 22:18:21.265888 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:21.265856 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:21.267099 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:21.267070 2580 patch_prober.go:28] interesting pod/e2e-derive-85902df0-openshift-default-7c87597755-592bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 21 22:18:21.267226 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:21.267130 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" podUID="9c9eee7f-ee88-4399-8684-1e83baa3e25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 21 22:18:22.266026 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:22.265985 2580 patch_prober.go:28] interesting pod/e2e-derive-85902df0-openshift-default-7c87597755-592bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 21 22:18:22.266425 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:22.266046 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" podUID="9c9eee7f-ee88-4399-8684-1e83baa3e25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 21 22:18:23.266458 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:23.266421 2580 patch_prober.go:28] interesting pod/e2e-derive-85902df0-openshift-default-7c87597755-592bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 21 22:18:23.266802 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:23.266492 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" podUID="9c9eee7f-ee88-4399-8684-1e83baa3e25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 21 22:18:23.945470 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:23.945440 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:24.266071 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:24.266043 2580 patch_prober.go:28] interesting pod/e2e-derive-85902df0-openshift-default-7c87597755-592bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 21 22:18:24.266236 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:24.266103 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" podUID="9c9eee7f-ee88-4399-8684-1e83baa3e25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 21 22:18:25.269746 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:25.269716 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:25.270759 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:25.270739 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:48.451810 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:48.451774 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm"] Jul 21 22:18:48.452259 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:48.452044 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" podUID="9c9eee7f-ee88-4399-8684-1e83baa3e25a" containerName="istio-proxy" containerID="cri-o://8cd8a02b66754baa424fc83faf417a00ed39d8c30d0b37702d1e4ea607415f5f" gracePeriod=30 Jul 21 22:18:53.717634 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.717577 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:53.738045 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.738015 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-envoy\") pod \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " Jul 21 22:18:53.738258 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.738053 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-podinfo\") pod \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " Jul 21 22:18:53.738258 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.738085 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-token\") pod \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " Jul 21 22:18:53.738258 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.738116 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shfg6\" (UniqueName: \"kubernetes.io/projected/9c9eee7f-ee88-4399-8684-1e83baa3e25a-kube-api-access-shfg6\") pod \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " Jul 21 22:18:53.738671 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.738646 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istiod-ca-cert\") pod \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " Jul 21 22:18:53.738759 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.738743 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-data\") pod \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " Jul 21 22:18:53.738822 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.738800 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-credential-socket\") pod \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " Jul 21 22:18:53.738887 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.738846 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-workload-socket\") pod \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " Jul 21 22:18:53.738887 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.738873 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-workload-certs\") pod \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\" (UID: \"9c9eee7f-ee88-4399-8684-1e83baa3e25a\") " Jul 21 22:18:53.739169 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.739130 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c9eee7f-ee88-4399-8684-1e83baa3e25a" (UID: "9c9eee7f-ee88-4399-8684-1e83baa3e25a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:18:53.739392 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.739340 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c9eee7f-ee88-4399-8684-1e83baa3e25a" (UID: "9c9eee7f-ee88-4399-8684-1e83baa3e25a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:18:53.739580 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.739542 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-workload-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:18:53.739767 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.739752 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istiod-ca-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:18:53.739899 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.739625 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c9eee7f-ee88-4399-8684-1e83baa3e25a" (UID: "9c9eee7f-ee88-4399-8684-1e83baa3e25a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:18:53.740008 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.739988 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c9eee7f-ee88-4399-8684-1e83baa3e25a" (UID: "9c9eee7f-ee88-4399-8684-1e83baa3e25a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:18:53.740114 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.740091 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c9eee7f-ee88-4399-8684-1e83baa3e25a" (UID: "9c9eee7f-ee88-4399-8684-1e83baa3e25a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:18:53.741459 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.741435 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c9eee7f-ee88-4399-8684-1e83baa3e25a" (UID: "9c9eee7f-ee88-4399-8684-1e83baa3e25a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:18:53.741726 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.741683 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c9eee7f-ee88-4399-8684-1e83baa3e25a" (UID: "9c9eee7f-ee88-4399-8684-1e83baa3e25a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:18:53.742002 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.741975 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c9eee7f-ee88-4399-8684-1e83baa3e25a" (UID: "9c9eee7f-ee88-4399-8684-1e83baa3e25a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 22:18:53.742315 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.742287 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9eee7f-ee88-4399-8684-1e83baa3e25a-kube-api-access-shfg6" (OuterVolumeSpecName: "kube-api-access-shfg6") pod "9c9eee7f-ee88-4399-8684-1e83baa3e25a" (UID: "9c9eee7f-ee88-4399-8684-1e83baa3e25a"). InnerVolumeSpecName "kube-api-access-shfg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:18:53.841152 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.841126 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-workload-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:18:53.841152 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.841149 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-envoy\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:18:53.841314 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.841158 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-podinfo\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:18:53.841314 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.841166 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-token\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:18:53.841314 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.841175 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shfg6\" (UniqueName: \"kubernetes.io/projected/9c9eee7f-ee88-4399-8684-1e83baa3e25a-kube-api-access-shfg6\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:18:53.841314 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.841183 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-istio-data\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:18:53.841314 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:53.841191 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9eee7f-ee88-4399-8684-1e83baa3e25a-credential-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:18:54.202583 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:54.202545 2580 generic.go:356] "Generic (PLEG): container finished" podID="9c9eee7f-ee88-4399-8684-1e83baa3e25a" containerID="8cd8a02b66754baa424fc83faf417a00ed39d8c30d0b37702d1e4ea607415f5f" exitCode=0 Jul 21 22:18:54.202787 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:54.202592 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" Jul 21 22:18:54.202787 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:54.202597 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" event={"ID":"9c9eee7f-ee88-4399-8684-1e83baa3e25a","Type":"ContainerDied","Data":"8cd8a02b66754baa424fc83faf417a00ed39d8c30d0b37702d1e4ea607415f5f"} Jul 21 22:18:54.202787 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:54.202649 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm" event={"ID":"9c9eee7f-ee88-4399-8684-1e83baa3e25a","Type":"ContainerDied","Data":"6d95db93ca7339a26578329ce32a6595a93f7f05c50fdf97f49c3f0846cff2ad"} Jul 21 22:18:54.202787 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:54.202677 2580 scope.go:117] "RemoveContainer" containerID="8cd8a02b66754baa424fc83faf417a00ed39d8c30d0b37702d1e4ea607415f5f" Jul 21 22:18:54.211946 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:54.211930 2580 scope.go:117] "RemoveContainer" containerID="41513cdc74313c2498a3d4c8629da4c40fdf60ad429e72608046dec3f9923024" Jul 21 22:18:54.219762 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:54.219739 2580 scope.go:117] "RemoveContainer" containerID="8cd8a02b66754baa424fc83faf417a00ed39d8c30d0b37702d1e4ea607415f5f" Jul 21 22:18:54.220016 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:18:54.219999 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd8a02b66754baa424fc83faf417a00ed39d8c30d0b37702d1e4ea607415f5f\": container with ID starting with 8cd8a02b66754baa424fc83faf417a00ed39d8c30d0b37702d1e4ea607415f5f not found: ID does not exist" containerID="8cd8a02b66754baa424fc83faf417a00ed39d8c30d0b37702d1e4ea607415f5f" Jul 21 22:18:54.220068 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:54.220027 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd8a02b66754baa424fc83faf417a00ed39d8c30d0b37702d1e4ea607415f5f"} err="failed to get container status \"8cd8a02b66754baa424fc83faf417a00ed39d8c30d0b37702d1e4ea607415f5f\": rpc error: code = NotFound desc = could not find container \"8cd8a02b66754baa424fc83faf417a00ed39d8c30d0b37702d1e4ea607415f5f\": container with ID starting with 8cd8a02b66754baa424fc83faf417a00ed39d8c30d0b37702d1e4ea607415f5f not found: ID does not exist" Jul 21 22:18:54.220068 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:54.220042 2580 scope.go:117] "RemoveContainer" containerID="41513cdc74313c2498a3d4c8629da4c40fdf60ad429e72608046dec3f9923024" Jul 21 22:18:54.220268 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:18:54.220247 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41513cdc74313c2498a3d4c8629da4c40fdf60ad429e72608046dec3f9923024\": container with ID starting with 41513cdc74313c2498a3d4c8629da4c40fdf60ad429e72608046dec3f9923024 not found: ID does not exist" containerID="41513cdc74313c2498a3d4c8629da4c40fdf60ad429e72608046dec3f9923024" Jul 21 22:18:54.220310 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:54.220274 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41513cdc74313c2498a3d4c8629da4c40fdf60ad429e72608046dec3f9923024"} err="failed to get container status \"41513cdc74313c2498a3d4c8629da4c40fdf60ad429e72608046dec3f9923024\": rpc error: code = NotFound desc = could not find container \"41513cdc74313c2498a3d4c8629da4c40fdf60ad429e72608046dec3f9923024\": container with ID starting with 41513cdc74313c2498a3d4c8629da4c40fdf60ad429e72608046dec3f9923024 not found: ID does not exist" Jul 21 22:18:54.225751 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:54.225730 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm"] Jul 21 22:18:54.229590 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:54.229571 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-85902df0-openshift-default-7c87597755-592bm"] Jul 21 22:18:54.779721 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:18:54.779688 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c9eee7f-ee88-4399-8684-1e83baa3e25a" path="/var/lib/kubelet/pods/9c9eee7f-ee88-4399-8684-1e83baa3e25a/volumes" Jul 21 22:20:28.015309 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.015273 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9"] Jul 21 22:20:28.015857 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.015754 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c9eee7f-ee88-4399-8684-1e83baa3e25a" containerName="istio-proxy" Jul 21 22:20:28.015857 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.015767 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c9eee7f-ee88-4399-8684-1e83baa3e25a" containerName="istio-proxy" Jul 21 22:20:28.015857 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.015776 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c9eee7f-ee88-4399-8684-1e83baa3e25a" containerName="istio-proxy" Jul 21 22:20:28.015857 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.015782 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c9eee7f-ee88-4399-8684-1e83baa3e25a" containerName="istio-proxy" Jul 21 22:20:28.015857 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.015803 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8701da77-52b8-489b-9331-df57ecdac65e" containerName="istio-proxy" Jul 21 22:20:28.015857 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.015809 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8701da77-52b8-489b-9331-df57ecdac65e" containerName="istio-proxy" Jul 21 22:20:28.015857 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.015822 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8701da77-52b8-489b-9331-df57ecdac65e" containerName="istio-proxy" Jul 21 22:20:28.015857 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.015827 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8701da77-52b8-489b-9331-df57ecdac65e" containerName="istio-proxy" Jul 21 22:20:28.016142 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.015884 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c9eee7f-ee88-4399-8684-1e83baa3e25a" containerName="istio-proxy" Jul 21 22:20:28.016142 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.015891 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8701da77-52b8-489b-9331-df57ecdac65e" containerName="istio-proxy" Jul 21 22:20:28.016142 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.015900 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8701da77-52b8-489b-9331-df57ecdac65e" containerName="istio-proxy" Jul 21 22:20:28.016142 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.015907 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c9eee7f-ee88-4399-8684-1e83baa3e25a" containerName="istio-proxy" Jul 21 22:20:28.018959 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.018943 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.021818 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.021793 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e0aec917-openshift-default-dockercfg-9knwh\"" Jul 21 22:20:28.041031 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.041005 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9"] Jul 21 22:20:28.158169 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.158138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istiod-ca-cert\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.158169 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.158172 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-workload-socket\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.158437 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.158258 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-workload-certs\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.158437 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.158297 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-credential-socket\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.158437 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.158352 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-data\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.158558 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.158437 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvwwt\" (UniqueName: \"kubernetes.io/projected/800fcf22-d829-4298-99b4-11a2b7b3e0f0-kube-api-access-dvwwt\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.158558 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.158470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-token\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.158558 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.158495 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-envoy\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.158558 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.158534 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-podinfo\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.259860 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.259828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-credential-socket\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.260024 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.259866 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-data\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.260024 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.259908 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvwwt\" (UniqueName: \"kubernetes.io/projected/800fcf22-d829-4298-99b4-11a2b7b3e0f0-kube-api-access-dvwwt\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.260024 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.259943 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-token\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.260024 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.259973 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-envoy\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.260024 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.260000 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-podinfo\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.260297 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.260056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istiod-ca-cert\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.260297 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.260081 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-workload-socket\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.260297 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.260162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-workload-certs\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.260297 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.260274 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-data\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.260527 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.260331 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-credential-socket\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.260589 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.260545 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-workload-socket\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.260644 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.260589 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-workload-certs\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.261032 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.260988 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istiod-ca-cert\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.262477 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.262455 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-envoy\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.262675 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.262657 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-podinfo\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.272143 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.272073 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-token\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.272248 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.272197 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvwwt\" (UniqueName: \"kubernetes.io/projected/800fcf22-d829-4298-99b4-11a2b7b3e0f0-kube-api-access-dvwwt\") pod \"e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.329607 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.329576 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:28.472438 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.472411 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9"] Jul 21 22:20:28.473990 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:20:28.473962 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800fcf22_d829_4298_99b4_11a2b7b3e0f0.slice/crio-d36a04a6d06ac48ef1b7e857d94d55d9dfcb6da7f57aacf43806ed899b9a0bcb WatchSource:0}: Error finding container d36a04a6d06ac48ef1b7e857d94d55d9dfcb6da7f57aacf43806ed899b9a0bcb: Status 404 returned error can't find the container with id d36a04a6d06ac48ef1b7e857d94d55d9dfcb6da7f57aacf43806ed899b9a0bcb Jul 21 22:20:28.476495 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.476459 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:20:28.476590 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.476547 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:20:28.476659 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.476592 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:20:28.710566 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.710521 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" event={"ID":"800fcf22-d829-4298-99b4-11a2b7b3e0f0","Type":"ContainerStarted","Data":"0c55a9be47ebe2d47d251357f14e824214fd354834f74fc70bb71971777b9717"} Jul 21 22:20:28.710767 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.710583 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" event={"ID":"800fcf22-d829-4298-99b4-11a2b7b3e0f0","Type":"ContainerStarted","Data":"d36a04a6d06ac48ef1b7e857d94d55d9dfcb6da7f57aacf43806ed899b9a0bcb"} Jul 21 22:20:28.734021 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:28.733962 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" podStartSLOduration=1.7339435920000001 podStartE2EDuration="1.733943592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:20:28.730472043 +0000 UTC m=+1906.493706305" watchObservedRunningTime="2026-07-21 22:20:28.733943592 +0000 UTC m=+1906.497177856" Jul 21 22:20:29.330255 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:29.330225 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:29.335402 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:29.335355 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:29.715722 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:29.715639 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:29.716464 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:29.716445 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:42.789426 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:42.789395 2580 generic.go:356] "Generic (PLEG): container finished" podID="800fcf22-d829-4298-99b4-11a2b7b3e0f0" containerID="0c55a9be47ebe2d47d251357f14e824214fd354834f74fc70bb71971777b9717" exitCode=0 Jul 21 22:20:42.789832 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:42.789457 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" event={"ID":"800fcf22-d829-4298-99b4-11a2b7b3e0f0","Type":"ContainerDied","Data":"0c55a9be47ebe2d47d251357f14e824214fd354834f74fc70bb71971777b9717"} Jul 21 22:20:42.789880 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:42.789830 2580 scope.go:117] "RemoveContainer" containerID="0c55a9be47ebe2d47d251357f14e824214fd354834f74fc70bb71971777b9717" Jul 21 22:20:42.790885 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:42.790852 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:20:42.790957 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:42.790939 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:20:42.790997 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:42.790981 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:20:43.801360 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:43.801319 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" event={"ID":"800fcf22-d829-4298-99b4-11a2b7b3e0f0","Type":"ContainerStarted","Data":"dbd6408c16dde301233cefecb7ee86207686827a050544df28a60c95039e792d"} Jul 21 22:20:44.329910 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:44.329877 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:44.330942 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:44.330918 2580 patch_prober.go:28] interesting pod/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 21 22:20:44.331060 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:44.330968 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" podUID="800fcf22-d829-4298-99b4-11a2b7b3e0f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 21 22:20:44.715869 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:44.715789 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:45.330915 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:45.330881 2580 patch_prober.go:28] interesting pod/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 21 22:20:45.331308 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:45.330936 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" podUID="800fcf22-d829-4298-99b4-11a2b7b3e0f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 21 22:20:46.330439 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:46.330408 2580 patch_prober.go:28] interesting pod/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 21 22:20:46.330604 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:46.330456 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" podUID="800fcf22-d829-4298-99b4-11a2b7b3e0f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 21 22:20:47.330951 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:47.330914 2580 patch_prober.go:28] interesting pod/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 21 22:20:47.331318 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:47.330977 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" podUID="800fcf22-d829-4298-99b4-11a2b7b3e0f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 21 22:20:48.334027 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:48.333998 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:20:48.334959 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:20:48.334938 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:21:04.813879 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:04.813847 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9"] Jul 21 22:21:04.814354 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:04.814048 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" podUID="800fcf22-d829-4298-99b4-11a2b7b3e0f0" containerName="istio-proxy" containerID="cri-o://dbd6408c16dde301233cefecb7ee86207686827a050544df28a60c95039e792d" gracePeriod=30 Jul 21 22:21:09.944852 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:09.944818 2580 generic.go:356] "Generic (PLEG): container finished" podID="800fcf22-d829-4298-99b4-11a2b7b3e0f0" containerID="dbd6408c16dde301233cefecb7ee86207686827a050544df28a60c95039e792d" exitCode=0 Jul 21 22:21:09.945265 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:09.944888 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" event={"ID":"800fcf22-d829-4298-99b4-11a2b7b3e0f0","Type":"ContainerDied","Data":"dbd6408c16dde301233cefecb7ee86207686827a050544df28a60c95039e792d"} Jul 21 22:21:09.945265 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:09.944950 2580 scope.go:117] "RemoveContainer" containerID="0c55a9be47ebe2d47d251357f14e824214fd354834f74fc70bb71971777b9717" Jul 21 22:21:10.067933 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.067906 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:21:10.177708 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.177618 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-data\") pod \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " Jul 21 22:21:10.177708 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.177702 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-credential-socket\") pod \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " Jul 21 22:21:10.177944 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.177730 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-podinfo\") pod \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " Jul 21 22:21:10.177944 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.177757 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-workload-certs\") pod \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " Jul 21 22:21:10.177944 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.177782 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-token\") pod \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " Jul 21 22:21:10.177944 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.177822 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istiod-ca-cert\") pod \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " Jul 21 22:21:10.177944 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.177906 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-workload-socket\") pod \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " Jul 21 22:21:10.178199 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.177969 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-envoy\") pod \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " Jul 21 22:21:10.178199 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.178025 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvwwt\" (UniqueName: \"kubernetes.io/projected/800fcf22-d829-4298-99b4-11a2b7b3e0f0-kube-api-access-dvwwt\") pod \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\" (UID: \"800fcf22-d829-4298-99b4-11a2b7b3e0f0\") " Jul 21 22:21:10.178199 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.178052 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "800fcf22-d829-4298-99b4-11a2b7b3e0f0" (UID: "800fcf22-d829-4298-99b4-11a2b7b3e0f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:21:10.178199 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.178076 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "800fcf22-d829-4298-99b4-11a2b7b3e0f0" (UID: "800fcf22-d829-4298-99b4-11a2b7b3e0f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:21:10.178445 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.178265 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "800fcf22-d829-4298-99b4-11a2b7b3e0f0" (UID: "800fcf22-d829-4298-99b4-11a2b7b3e0f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:21:10.178445 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.178315 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "800fcf22-d829-4298-99b4-11a2b7b3e0f0" (UID: "800fcf22-d829-4298-99b4-11a2b7b3e0f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:21:10.178445 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.178437 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-credential-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:21:10.178584 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.178457 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-workload-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:21:10.178584 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.178471 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istiod-ca-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:21:10.178584 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.178484 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-workload-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:21:10.178692 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.178584 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-data" (OuterVolumeSpecName: "istio-data") pod "800fcf22-d829-4298-99b4-11a2b7b3e0f0" (UID: "800fcf22-d829-4298-99b4-11a2b7b3e0f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:21:10.180316 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.180292 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800fcf22-d829-4298-99b4-11a2b7b3e0f0-kube-api-access-dvwwt" (OuterVolumeSpecName: "kube-api-access-dvwwt") pod "800fcf22-d829-4298-99b4-11a2b7b3e0f0" (UID: "800fcf22-d829-4298-99b4-11a2b7b3e0f0"). InnerVolumeSpecName "kube-api-access-dvwwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:21:10.180424 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.180318 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "800fcf22-d829-4298-99b4-11a2b7b3e0f0" (UID: "800fcf22-d829-4298-99b4-11a2b7b3e0f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:21:10.180607 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.180591 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-token" (OuterVolumeSpecName: "istio-token") pod "800fcf22-d829-4298-99b4-11a2b7b3e0f0" (UID: "800fcf22-d829-4298-99b4-11a2b7b3e0f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:21:10.180666 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.180647 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "800fcf22-d829-4298-99b4-11a2b7b3e0f0" (UID: "800fcf22-d829-4298-99b4-11a2b7b3e0f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 22:21:10.279365 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.279327 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-data\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:21:10.279365 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.279360 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-podinfo\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:21:10.279365 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.279399 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-token\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:21:10.279638 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.279413 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800fcf22-d829-4298-99b4-11a2b7b3e0f0-istio-envoy\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:21:10.279638 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.279425 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvwwt\" (UniqueName: \"kubernetes.io/projected/800fcf22-d829-4298-99b4-11a2b7b3e0f0-kube-api-access-dvwwt\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:21:10.958564 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.958524 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" event={"ID":"800fcf22-d829-4298-99b4-11a2b7b3e0f0","Type":"ContainerDied","Data":"d36a04a6d06ac48ef1b7e857d94d55d9dfcb6da7f57aacf43806ed899b9a0bcb"} Jul 21 22:21:10.959068 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.958617 2580 scope.go:117] "RemoveContainer" containerID="dbd6408c16dde301233cefecb7ee86207686827a050544df28a60c95039e792d" Jul 21 22:21:10.959068 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.958543 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9" Jul 21 22:21:10.979504 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.979468 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9"] Jul 21 22:21:10.986529 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:10.986497 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e0aec917-openshift-default-6bcc45f847-dsdg9"] Jul 21 22:21:12.779542 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:12.779506 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="800fcf22-d829-4298-99b4-11a2b7b3e0f0" path="/var/lib/kubelet/pods/800fcf22-d829-4298-99b4-11a2b7b3e0f0/volumes" Jul 21 22:21:25.424327 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.424293 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv"] Jul 21 22:21:25.424771 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.424757 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="800fcf22-d829-4298-99b4-11a2b7b3e0f0" containerName="istio-proxy" Jul 21 22:21:25.424818 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.424773 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="800fcf22-d829-4298-99b4-11a2b7b3e0f0" containerName="istio-proxy" Jul 21 22:21:25.424868 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.424859 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="800fcf22-d829-4298-99b4-11a2b7b3e0f0" containerName="istio-proxy" Jul 21 22:21:25.428106 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.428090 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.430774 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.430750 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6699167f-openshift-default-dockercfg-gqh2t\"" Jul 21 22:21:25.439175 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.439150 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv"] Jul 21 22:21:25.522866 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.522817 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/564ed43f-1443-403d-8958-4b0267a76221-istio-podinfo\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.523063 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.522881 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-workload-certs\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.523063 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.522914 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-istio-envoy\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.523063 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.522939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-istio-data\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.523063 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.523035 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bczz\" (UniqueName: \"kubernetes.io/projected/564ed43f-1443-403d-8958-4b0267a76221-kube-api-access-6bczz\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.523260 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.523079 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/564ed43f-1443-403d-8958-4b0267a76221-istio-token\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.523260 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.523120 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-workload-socket\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.523260 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.523197 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-credential-socket\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.523260 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.523221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/564ed43f-1443-403d-8958-4b0267a76221-istiod-ca-cert\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.624564 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.624506 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-credential-socket\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.624564 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.624564 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/564ed43f-1443-403d-8958-4b0267a76221-istiod-ca-cert\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.624831 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.624614 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/564ed43f-1443-403d-8958-4b0267a76221-istio-podinfo\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.624831 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.624641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-workload-certs\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.624831 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.624663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-istio-envoy\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.624831 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.624681 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-istio-data\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.624831 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.624730 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bczz\" (UniqueName: \"kubernetes.io/projected/564ed43f-1443-403d-8958-4b0267a76221-kube-api-access-6bczz\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.624831 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.624759 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/564ed43f-1443-403d-8958-4b0267a76221-istio-token\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.624831 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.624785 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-workload-socket\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.625174 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.625021 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-workload-certs\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.625174 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.625103 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-credential-socket\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.625292 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.625251 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-istio-data\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.625292 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.625268 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-workload-socket\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.625425 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.625366 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/564ed43f-1443-403d-8958-4b0267a76221-istiod-ca-cert\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.627127 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.627110 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-istio-envoy\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.627351 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.627330 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/564ed43f-1443-403d-8958-4b0267a76221-istio-podinfo\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.633551 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.633518 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/564ed43f-1443-403d-8958-4b0267a76221-istio-token\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.633670 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.633657 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bczz\" (UniqueName: \"kubernetes.io/projected/564ed43f-1443-403d-8958-4b0267a76221-kube-api-access-6bczz\") pod \"e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.739365 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.739334 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:25.885776 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.885747 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv"] Jul 21 22:21:25.888050 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:21:25.888024 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564ed43f_1443_403d_8958_4b0267a76221.slice/crio-67839d7c7a6e926f794a2dbd5fc42fe49c191c2a2e54c04dd565a96ab6234b6e WatchSource:0}: Error finding container 67839d7c7a6e926f794a2dbd5fc42fe49c191c2a2e54c04dd565a96ab6234b6e: Status 404 returned error can't find the container with id 67839d7c7a6e926f794a2dbd5fc42fe49c191c2a2e54c04dd565a96ab6234b6e Jul 21 22:21:25.890221 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.890187 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:21:25.890313 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.890259 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:21:25.890313 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:25.890293 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:21:26.040698 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:26.040607 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" event={"ID":"564ed43f-1443-403d-8958-4b0267a76221","Type":"ContainerStarted","Data":"50fdc380b8fcd465b552c52ff71ea7019e0aaf7b3406f7bc407ed8b4ed541e2d"} Jul 21 22:21:26.040698 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:26.040664 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" event={"ID":"564ed43f-1443-403d-8958-4b0267a76221","Type":"ContainerStarted","Data":"67839d7c7a6e926f794a2dbd5fc42fe49c191c2a2e54c04dd565a96ab6234b6e"} Jul 21 22:21:26.064002 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:26.063943 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" podStartSLOduration=1.063926647 podStartE2EDuration="1.063926647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:21:26.061187373 +0000 UTC m=+1963.824421647" watchObservedRunningTime="2026-07-21 22:21:26.063926647 +0000 UTC m=+1963.827160907" Jul 21 22:21:26.739574 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:26.739528 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:26.744589 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:26.744566 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:27.046212 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:27.046132 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:27.047054 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:27.047039 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:39.123562 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:39.123513 2580 generic.go:356] "Generic (PLEG): container finished" podID="564ed43f-1443-403d-8958-4b0267a76221" containerID="50fdc380b8fcd465b552c52ff71ea7019e0aaf7b3406f7bc407ed8b4ed541e2d" exitCode=0 Jul 21 22:21:39.124008 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:39.123589 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" event={"ID":"564ed43f-1443-403d-8958-4b0267a76221","Type":"ContainerDied","Data":"50fdc380b8fcd465b552c52ff71ea7019e0aaf7b3406f7bc407ed8b4ed541e2d"} Jul 21 22:21:39.124065 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:39.124024 2580 scope.go:117] "RemoveContainer" containerID="50fdc380b8fcd465b552c52ff71ea7019e0aaf7b3406f7bc407ed8b4ed541e2d" Jul 21 22:21:39.125107 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:39.125073 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:21:39.125188 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:39.125163 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:21:39.125230 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:39.125221 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:21:40.130896 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:40.130859 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" event={"ID":"564ed43f-1443-403d-8958-4b0267a76221","Type":"ContainerStarted","Data":"0f8384c7ab273311768c02ce1de6ba750411409aab34be9b11cda7d18948e7a2"} Jul 21 22:21:40.740406 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:40.740358 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:40.741516 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:40.741491 2580 patch_prober.go:28] interesting pod/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 21 22:21:40.741636 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:40.741540 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" podUID="564ed43f-1443-403d-8958-4b0267a76221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 21 22:21:41.739834 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:41.739794 2580 patch_prober.go:28] interesting pod/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 21 22:21:41.740225 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:41.739864 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" podUID="564ed43f-1443-403d-8958-4b0267a76221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 21 22:21:42.046554 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:42.046450 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:42.740354 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:42.740314 2580 patch_prober.go:28] interesting pod/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 21 22:21:42.740794 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:42.740411 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" podUID="564ed43f-1443-403d-8958-4b0267a76221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 21 22:21:43.740638 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:43.740601 2580 patch_prober.go:28] interesting pod/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 21 22:21:43.741036 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:43.740673 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" podUID="564ed43f-1443-403d-8958-4b0267a76221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 21 22:21:44.743978 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:44.743949 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:21:44.745203 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:21:44.745174 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:23:22.444269 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:22.444235 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv"] Jul 21 22:23:22.444840 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:22.444493 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" podUID="564ed43f-1443-403d-8958-4b0267a76221" containerName="istio-proxy" containerID="cri-o://0f8384c7ab273311768c02ce1de6ba750411409aab34be9b11cda7d18948e7a2" gracePeriod=30 Jul 21 22:23:27.699651 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.699617 2580 generic.go:356] "Generic (PLEG): container finished" podID="564ed43f-1443-403d-8958-4b0267a76221" containerID="0f8384c7ab273311768c02ce1de6ba750411409aab34be9b11cda7d18948e7a2" exitCode=0 Jul 21 22:23:27.700056 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.699690 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" event={"ID":"564ed43f-1443-403d-8958-4b0267a76221","Type":"ContainerDied","Data":"0f8384c7ab273311768c02ce1de6ba750411409aab34be9b11cda7d18948e7a2"} Jul 21 22:23:27.700056 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.699765 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" event={"ID":"564ed43f-1443-403d-8958-4b0267a76221","Type":"ContainerDied","Data":"67839d7c7a6e926f794a2dbd5fc42fe49c191c2a2e54c04dd565a96ab6234b6e"} Jul 21 22:23:27.700056 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.699795 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67839d7c7a6e926f794a2dbd5fc42fe49c191c2a2e54c04dd565a96ab6234b6e" Jul 21 22:23:27.700056 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.699816 2580 scope.go:117] "RemoveContainer" containerID="50fdc380b8fcd465b552c52ff71ea7019e0aaf7b3406f7bc407ed8b4ed541e2d" Jul 21 22:23:27.707551 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.707506 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:23:27.869644 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.869609 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-istio-envoy\") pod \"564ed43f-1443-403d-8958-4b0267a76221\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " Jul 21 22:23:27.869829 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.869661 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-workload-certs\") pod \"564ed43f-1443-403d-8958-4b0267a76221\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " Jul 21 22:23:27.869829 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.869698 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-istio-data\") pod \"564ed43f-1443-403d-8958-4b0267a76221\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " Jul 21 22:23:27.869829 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.869728 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/564ed43f-1443-403d-8958-4b0267a76221-istio-podinfo\") pod \"564ed43f-1443-403d-8958-4b0267a76221\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " Jul 21 22:23:27.869829 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.869789 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-credential-socket\") pod \"564ed43f-1443-403d-8958-4b0267a76221\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " Jul 21 22:23:27.870035 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.869829 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/564ed43f-1443-403d-8958-4b0267a76221-istio-token\") pod \"564ed43f-1443-403d-8958-4b0267a76221\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " Jul 21 22:23:27.870035 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.869886 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/564ed43f-1443-403d-8958-4b0267a76221-istiod-ca-cert\") pod \"564ed43f-1443-403d-8958-4b0267a76221\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " Jul 21 22:23:27.870035 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.869917 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bczz\" (UniqueName: \"kubernetes.io/projected/564ed43f-1443-403d-8958-4b0267a76221-kube-api-access-6bczz\") pod \"564ed43f-1443-403d-8958-4b0267a76221\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " Jul 21 22:23:27.870035 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.869939 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-workload-socket\") pod \"564ed43f-1443-403d-8958-4b0267a76221\" (UID: \"564ed43f-1443-403d-8958-4b0267a76221\") " Jul 21 22:23:27.870253 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.870029 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "564ed43f-1443-403d-8958-4b0267a76221" (UID: "564ed43f-1443-403d-8958-4b0267a76221"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:23:27.870253 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.870194 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "564ed43f-1443-403d-8958-4b0267a76221" (UID: "564ed43f-1443-403d-8958-4b0267a76221"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:23:27.870253 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.870244 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "564ed43f-1443-403d-8958-4b0267a76221" (UID: "564ed43f-1443-403d-8958-4b0267a76221"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:23:27.870423 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.870270 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564ed43f-1443-403d-8958-4b0267a76221-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "564ed43f-1443-403d-8958-4b0267a76221" (UID: "564ed43f-1443-403d-8958-4b0267a76221"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:23:27.870539 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.870521 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-credential-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:23:27.870609 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.870544 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/564ed43f-1443-403d-8958-4b0267a76221-istiod-ca-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:23:27.870609 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.870558 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-workload-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:23:27.870609 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.870570 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-workload-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:23:27.870831 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.870804 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-istio-data" (OuterVolumeSpecName: "istio-data") pod "564ed43f-1443-403d-8958-4b0267a76221" (UID: "564ed43f-1443-403d-8958-4b0267a76221"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:23:27.872219 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.872178 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/564ed43f-1443-403d-8958-4b0267a76221-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "564ed43f-1443-403d-8958-4b0267a76221" (UID: "564ed43f-1443-403d-8958-4b0267a76221"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 22:23:27.872630 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.872611 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "564ed43f-1443-403d-8958-4b0267a76221" (UID: "564ed43f-1443-403d-8958-4b0267a76221"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:23:27.872753 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.872738 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564ed43f-1443-403d-8958-4b0267a76221-kube-api-access-6bczz" (OuterVolumeSpecName: "kube-api-access-6bczz") pod "564ed43f-1443-403d-8958-4b0267a76221" (UID: "564ed43f-1443-403d-8958-4b0267a76221"). InnerVolumeSpecName "kube-api-access-6bczz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:23:27.873050 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.873019 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564ed43f-1443-403d-8958-4b0267a76221-istio-token" (OuterVolumeSpecName: "istio-token") pod "564ed43f-1443-403d-8958-4b0267a76221" (UID: "564ed43f-1443-403d-8958-4b0267a76221"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:23:27.971439 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.971341 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-istio-data\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:23:27.971439 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.971368 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/564ed43f-1443-403d-8958-4b0267a76221-istio-podinfo\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:23:27.971439 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.971399 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/564ed43f-1443-403d-8958-4b0267a76221-istio-token\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:23:27.971439 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.971409 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bczz\" (UniqueName: \"kubernetes.io/projected/564ed43f-1443-403d-8958-4b0267a76221-kube-api-access-6bczz\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:23:27.971439 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:27.971418 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/564ed43f-1443-403d-8958-4b0267a76221-istio-envoy\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:23:28.705762 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:28.705730 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv" Jul 21 22:23:28.742448 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:28.742419 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv"] Jul 21 22:23:28.746651 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:28.746630 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6699167f-openshift-default-85d6f6ff6c-72flv"] Jul 21 22:23:28.779774 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:23:28.779743 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="564ed43f-1443-403d-8958-4b0267a76221" path="/var/lib/kubelet/pods/564ed43f-1443-403d-8958-4b0267a76221/volumes" Jul 21 22:24:18.833426 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.833367 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9"] Jul 21 22:24:18.833952 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.833917 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="800fcf22-d829-4298-99b4-11a2b7b3e0f0" containerName="istio-proxy" Jul 21 22:24:18.833952 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.833937 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="800fcf22-d829-4298-99b4-11a2b7b3e0f0" containerName="istio-proxy" Jul 21 22:24:18.834037 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.833956 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="564ed43f-1443-403d-8958-4b0267a76221" containerName="istio-proxy" Jul 21 22:24:18.834037 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.833963 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="564ed43f-1443-403d-8958-4b0267a76221" containerName="istio-proxy" Jul 21 22:24:18.834099 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.834067 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="564ed43f-1443-403d-8958-4b0267a76221" containerName="istio-proxy" Jul 21 22:24:18.834099 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.834077 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="564ed43f-1443-403d-8958-4b0267a76221" containerName="istio-proxy" Jul 21 22:24:18.834099 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.834083 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="800fcf22-d829-4298-99b4-11a2b7b3e0f0" containerName="istio-proxy" Jul 21 22:24:18.837225 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.837209 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.839935 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.839915 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3f5ae6-openshift-default-dockercfg-529j9\"" Jul 21 22:24:18.848795 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.848772 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9"] Jul 21 22:24:18.873240 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.873211 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-data\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.873455 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.873244 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-credential-socket\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.873455 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.873354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istiod-ca-cert\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.873455 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.873405 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-envoy\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.873455 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.873435 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-workload-socket\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.873704 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.873462 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-token\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.873704 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.873496 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-podinfo\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.873704 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.873542 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpdtg\" (UniqueName: \"kubernetes.io/projected/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-kube-api-access-zpdtg\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.873704 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.873565 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-workload-certs\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.974640 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.974597 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istiod-ca-cert\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.974640 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.974654 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-envoy\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.974914 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.974687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-workload-socket\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.974914 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.974714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-token\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.974914 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.974732 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-podinfo\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.974914 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.974752 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpdtg\" (UniqueName: \"kubernetes.io/projected/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-kube-api-access-zpdtg\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.974914 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.974791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-workload-certs\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.974914 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.974900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-data\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.975231 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.974928 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-credential-socket\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.975231 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.975152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-workload-socket\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.975342 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.975250 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-workload-certs\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.975342 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.975289 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-credential-socket\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.975479 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.975356 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-data\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.975542 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.975488 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istiod-ca-cert\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.977230 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.977202 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-envoy\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.977585 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.977569 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-podinfo\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.983287 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.983262 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpdtg\" (UniqueName: \"kubernetes.io/projected/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-kube-api-access-zpdtg\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:18.983420 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:18.983266 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-token\") pod \"e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:19.148677 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:19.148577 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:19.303426 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:19.303395 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9"] Jul 21 22:24:19.305023 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:24:19.304984 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab9a916_e8e9_4ce8_a9d1_a93e4abfe62a.slice/crio-35a105ad6efa5dc20ddf4ce7681e1e1ccfdffb1810966c4092db4b68b93e93d3 WatchSource:0}: Error finding container 35a105ad6efa5dc20ddf4ce7681e1e1ccfdffb1810966c4092db4b68b93e93d3: Status 404 returned error can't find the container with id 35a105ad6efa5dc20ddf4ce7681e1e1ccfdffb1810966c4092db4b68b93e93d3 Jul 21 22:24:19.307051 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:19.307011 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:24:19.307151 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:19.307096 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:24:19.307151 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:19.307139 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:24:19.968968 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:19.968930 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" event={"ID":"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a","Type":"ContainerStarted","Data":"cc9f0d8e1b588a568ea7663e4c6aceccba1111d37960970e2e0711a6639f1d64"} Jul 21 22:24:19.969393 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:19.968985 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" event={"ID":"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a","Type":"ContainerStarted","Data":"35a105ad6efa5dc20ddf4ce7681e1e1ccfdffb1810966c4092db4b68b93e93d3"} Jul 21 22:24:19.993361 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:19.993308 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" podStartSLOduration=1.993293929 podStartE2EDuration="1.993293929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:24:19.990435172 +0000 UTC m=+2137.753669460" watchObservedRunningTime="2026-07-21 22:24:19.993293929 +0000 UTC m=+2137.756528189" Jul 21 22:24:20.149309 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:20.149277 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:20.154599 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:20.154578 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:20.975491 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:20.975460 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:20.976514 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:20.976494 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:27.118098 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.118065 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j"] Jul 21 22:24:27.118728 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.118706 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="564ed43f-1443-403d-8958-4b0267a76221" containerName="istio-proxy" Jul 21 22:24:27.118819 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.118729 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="564ed43f-1443-403d-8958-4b0267a76221" containerName="istio-proxy" Jul 21 22:24:27.122231 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.122208 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.125443 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.125410 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4ad703-openshift-default-dockercfg-4cmx9\"" Jul 21 22:24:27.134981 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.134956 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j"] Jul 21 22:24:27.260240 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.260203 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-credential-socket\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.260451 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.260257 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-data\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.260451 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.260279 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-podinfo\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.260451 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.260354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istiod-ca-cert\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.260600 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.260490 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-token\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.260600 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.260517 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-workload-certs\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.260600 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.260536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-workload-socket\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.260782 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.260606 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh7s4\" (UniqueName: \"kubernetes.io/projected/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-kube-api-access-lh7s4\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.260782 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.260628 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-envoy\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.361568 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.361531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-token\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.361761 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.361578 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-workload-certs\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.361761 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.361607 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-workload-socket\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.361761 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.361665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh7s4\" (UniqueName: \"kubernetes.io/projected/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-kube-api-access-lh7s4\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.361761 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.361693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-envoy\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.361761 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.361736 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-credential-socket\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.362030 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.361776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-data\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.362030 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.361810 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-podinfo\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.362030 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.361840 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istiod-ca-cert\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.362030 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.361999 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-workload-certs\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.362269 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.362036 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-workload-socket\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.362269 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.362075 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-credential-socket\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.362269 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.362173 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-data\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.362607 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.362583 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istiod-ca-cert\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.364130 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.364110 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-envoy\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.364325 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.364304 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-podinfo\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.373867 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.373802 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-token\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.373978 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.373959 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh7s4\" (UniqueName: \"kubernetes.io/projected/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-kube-api-access-lh7s4\") pod \"e2e-inf-b-4ad703-openshift-default-87d74fc-p274j\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.434389 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.434353 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:27.596405 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.593980 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j"] Jul 21 22:24:27.597403 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.597069 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:24:27.597403 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.597168 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:24:27.597403 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:27.597214 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:24:28.018151 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:28.018118 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" event={"ID":"1ba33d73-a499-4d7f-abe9-fce4ccfec92e","Type":"ContainerStarted","Data":"18fb678378f084d4bf3dbfb4be649c015a56755a7892dccd27e78ab1c0c57713"} Jul 21 22:24:28.018364 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:28.018177 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" event={"ID":"1ba33d73-a499-4d7f-abe9-fce4ccfec92e","Type":"ContainerStarted","Data":"2b29aaada8d0929cb24987d958ccb7873df1b57e2396441c1fa73f234e3d1e98"} Jul 21 22:24:28.042097 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:28.042034 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" podStartSLOduration=1.042011381 podStartE2EDuration="1.042011381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:24:28.041558075 +0000 UTC m=+2145.804792373" watchObservedRunningTime="2026-07-21 22:24:28.042011381 +0000 UTC m=+2145.805245644" Jul 21 22:24:28.435603 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:28.435498 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:28.440574 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:28.440547 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:29.023935 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:29.023901 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:29.024944 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:29.024923 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:35.058454 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:35.058419 2580 generic.go:356] "Generic (PLEG): container finished" podID="fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" containerID="cc9f0d8e1b588a568ea7663e4c6aceccba1111d37960970e2e0711a6639f1d64" exitCode=0 Jul 21 22:24:35.058912 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:35.058491 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" event={"ID":"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a","Type":"ContainerDied","Data":"cc9f0d8e1b588a568ea7663e4c6aceccba1111d37960970e2e0711a6639f1d64"} Jul 21 22:24:35.058912 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:35.058893 2580 scope.go:117] "RemoveContainer" containerID="cc9f0d8e1b588a568ea7663e4c6aceccba1111d37960970e2e0711a6639f1d64" Jul 21 22:24:35.060029 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:35.060003 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:24:35.060090 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:35.060068 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:24:35.060129 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:35.060103 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:24:35.975852 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:35.975809 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:36.065649 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:36.065606 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" event={"ID":"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a","Type":"ContainerStarted","Data":"d7cc60f62776718e6b570c5a2e9e2318df377a9c9cf955a06f94f7a80ff4c589"} Jul 21 22:24:36.149601 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:36.149563 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:36.150503 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:36.150478 2580 patch_prober.go:28] interesting pod/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 21 22:24:36.150623 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:36.150537 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" podUID="fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 21 22:24:37.149950 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:37.149911 2580 patch_prober.go:28] interesting pod/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 21 22:24:37.150338 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:37.149980 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" podUID="fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 21 22:24:38.149802 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:38.149763 2580 patch_prober.go:28] interesting pod/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 21 22:24:38.149977 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:38.149817 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" podUID="fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 21 22:24:39.149976 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:39.149942 2580 patch_prober.go:28] interesting pod/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 21 22:24:39.150330 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:39.149994 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" podUID="fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 21 22:24:40.153893 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:40.153861 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:40.154292 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:40.154268 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:40.155070 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:40.155047 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:24:43.115032 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:43.114999 2580 generic.go:356] "Generic (PLEG): container finished" podID="1ba33d73-a499-4d7f-abe9-fce4ccfec92e" containerID="18fb678378f084d4bf3dbfb4be649c015a56755a7892dccd27e78ab1c0c57713" exitCode=0 Jul 21 22:24:43.115484 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:43.115073 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" event={"ID":"1ba33d73-a499-4d7f-abe9-fce4ccfec92e","Type":"ContainerDied","Data":"18fb678378f084d4bf3dbfb4be649c015a56755a7892dccd27e78ab1c0c57713"} Jul 21 22:24:43.115545 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:43.115519 2580 scope.go:117] "RemoveContainer" containerID="18fb678378f084d4bf3dbfb4be649c015a56755a7892dccd27e78ab1c0c57713" Jul 21 22:24:43.116626 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:43.116597 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:24:43.116697 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:43.116668 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:24:43.116734 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:43.116703 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:24:44.024463 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:44.024419 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:44.122087 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:44.122051 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" event={"ID":"1ba33d73-a499-4d7f-abe9-fce4ccfec92e","Type":"ContainerStarted","Data":"aee4f00dfcc0d2b11e16da1a82e2b54d59a8f5add8c828767356bddca20e388f"} Jul 21 22:24:44.434645 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:44.434541 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:44.436079 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:44.436048 2580 patch_prober.go:28] interesting pod/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 21 22:24:44.436217 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:44.436105 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" podUID="1ba33d73-a499-4d7f-abe9-fce4ccfec92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 21 22:24:45.435439 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:45.435368 2580 patch_prober.go:28] interesting pod/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 21 22:24:45.435856 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:45.435460 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" podUID="1ba33d73-a499-4d7f-abe9-fce4ccfec92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 21 22:24:46.435612 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:46.435575 2580 patch_prober.go:28] interesting pod/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 21 22:24:46.436004 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:46.435644 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" podUID="1ba33d73-a499-4d7f-abe9-fce4ccfec92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 21 22:24:47.435519 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:47.435475 2580 patch_prober.go:28] interesting pod/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 21 22:24:47.435719 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:47.435556 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" podUID="1ba33d73-a499-4d7f-abe9-fce4ccfec92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 21 22:24:48.435038 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:48.434994 2580 patch_prober.go:28] interesting pod/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 21 22:24:48.435225 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:48.435055 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" podUID="1ba33d73-a499-4d7f-abe9-fce4ccfec92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 21 22:24:49.439211 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:49.439180 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:49.439725 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:49.439608 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:24:49.440431 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:24:49.440411 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:25:35.588777 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.588740 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9"] Jul 21 22:25:35.592887 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.592864 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.595868 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.595841 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4ad703\"/\"default-dockercfg-j7wlh\"" Jul 21 22:25:35.595868 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.595857 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4ad703\"/\"openshift-service-ca.crt\"" Jul 21 22:25:35.596054 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.595845 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4ad703\"/\"kube-root-ca.crt\"" Jul 21 22:25:35.596054 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.595887 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4ad703\"/\"test-model-4ad703-kserve-self-signed-certs\"" Jul 21 22:25:35.602408 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.602365 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9"] Jul 21 22:25:35.705489 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.705459 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-dshm\") pod \"test-model-4ad703-kserve-56bb5c76d7-jqfv9\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.705669 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.705500 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-model-cache\") pod \"test-model-4ad703-kserve-56bb5c76d7-jqfv9\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.705669 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.705525 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-tmp-dir\") pod \"test-model-4ad703-kserve-56bb5c76d7-jqfv9\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.705669 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.705609 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4wpq\" (UniqueName: \"kubernetes.io/projected/a037ac65-85de-46ce-8866-014de42ed204-kube-api-access-x4wpq\") pod \"test-model-4ad703-kserve-56bb5c76d7-jqfv9\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.705793 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.705676 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a037ac65-85de-46ce-8866-014de42ed204-tls-certs\") pod \"test-model-4ad703-kserve-56bb5c76d7-jqfv9\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.705793 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.705722 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-home\") pod \"test-model-4ad703-kserve-56bb5c76d7-jqfv9\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.806793 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.806765 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-home\") pod \"test-model-4ad703-kserve-56bb5c76d7-jqfv9\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.806941 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.806820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-dshm\") pod \"test-model-4ad703-kserve-56bb5c76d7-jqfv9\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.806941 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.806849 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-model-cache\") pod \"test-model-4ad703-kserve-56bb5c76d7-jqfv9\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.806941 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.806868 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-tmp-dir\") pod \"test-model-4ad703-kserve-56bb5c76d7-jqfv9\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.806941 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.806897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4wpq\" (UniqueName: \"kubernetes.io/projected/a037ac65-85de-46ce-8866-014de42ed204-kube-api-access-x4wpq\") pod \"test-model-4ad703-kserve-56bb5c76d7-jqfv9\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.807139 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.807047 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a037ac65-85de-46ce-8866-014de42ed204-tls-certs\") pod \"test-model-4ad703-kserve-56bb5c76d7-jqfv9\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.807239 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.807216 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-home\") pod \"test-model-4ad703-kserve-56bb5c76d7-jqfv9\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.807289 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.807254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-tmp-dir\") pod \"test-model-4ad703-kserve-56bb5c76d7-jqfv9\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.807289 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.807279 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-model-cache\") pod \"test-model-4ad703-kserve-56bb5c76d7-jqfv9\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.809264 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.809245 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-dshm\") pod \"test-model-4ad703-kserve-56bb5c76d7-jqfv9\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.809660 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.809641 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a037ac65-85de-46ce-8866-014de42ed204-tls-certs\") pod \"test-model-4ad703-kserve-56bb5c76d7-jqfv9\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.816867 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.816845 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4wpq\" (UniqueName: \"kubernetes.io/projected/a037ac65-85de-46ce-8866-014de42ed204-kube-api-access-x4wpq\") pod \"test-model-4ad703-kserve-56bb5c76d7-jqfv9\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:35.904194 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:35.904097 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:36.046273 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:36.046246 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9"] Jul 21 22:25:36.048607 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:25:36.048568 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda037ac65_85de_46ce_8866_014de42ed204.slice/crio-136c30c9dca8db6c55a858bf9f9ddd64398184c8f8440460b333256c3e1c5bf6 WatchSource:0}: Error finding container 136c30c9dca8db6c55a858bf9f9ddd64398184c8f8440460b333256c3e1c5bf6: Status 404 returned error can't find the container with id 136c30c9dca8db6c55a858bf9f9ddd64398184c8f8440460b333256c3e1c5bf6 Jul 21 22:25:36.408635 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:36.408596 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" event={"ID":"a037ac65-85de-46ce-8866-014de42ed204","Type":"ContainerStarted","Data":"eceb6778612ddd87126c5a40e55b573fb49c59cebf4dd6f6be8a656523e7536a"} Jul 21 22:25:36.408823 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:36.408646 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" event={"ID":"a037ac65-85de-46ce-8866-014de42ed204","Type":"ContainerStarted","Data":"136c30c9dca8db6c55a858bf9f9ddd64398184c8f8440460b333256c3e1c5bf6"} Jul 21 22:25:36.426700 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:36.426624 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" podStartSLOduration=1.426610433 podStartE2EDuration="1.426610433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:25:36.425633131 +0000 UTC m=+2214.188867394" watchObservedRunningTime="2026-07-21 22:25:36.426610433 +0000 UTC m=+2214.189844719" Jul 21 22:25:45.907938 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:45.907892 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:45.908540 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:45.907954 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:45.920428 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:45.920403 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:25:46.473088 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:25:46.473063 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:26:21.064393 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:21.064346 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9"] Jul 21 22:26:36.085481 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.085416 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" podUID="a037ac65-85de-46ce-8866-014de42ed204" containerName="main" containerID="cri-o://eceb6778612ddd87126c5a40e55b573fb49c59cebf4dd6f6be8a656523e7536a" gracePeriod=45 Jul 21 22:26:36.353520 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.353489 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:26:36.423771 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.423747 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-tmp-dir\") pod \"a037ac65-85de-46ce-8866-014de42ed204\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " Jul 21 22:26:36.423950 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.423781 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-dshm\") pod \"a037ac65-85de-46ce-8866-014de42ed204\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " Jul 21 22:26:36.423950 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.423805 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-home\") pod \"a037ac65-85de-46ce-8866-014de42ed204\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " Jul 21 22:26:36.423950 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.423835 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-model-cache\") pod \"a037ac65-85de-46ce-8866-014de42ed204\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " Jul 21 22:26:36.423950 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.423900 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4wpq\" (UniqueName: \"kubernetes.io/projected/a037ac65-85de-46ce-8866-014de42ed204-kube-api-access-x4wpq\") pod \"a037ac65-85de-46ce-8866-014de42ed204\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " Jul 21 22:26:36.424179 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.423955 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a037ac65-85de-46ce-8866-014de42ed204-tls-certs\") pod \"a037ac65-85de-46ce-8866-014de42ed204\" (UID: \"a037ac65-85de-46ce-8866-014de42ed204\") " Jul 21 22:26:36.424179 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.424039 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a037ac65-85de-46ce-8866-014de42ed204" (UID: "a037ac65-85de-46ce-8866-014de42ed204"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:26:36.424179 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.424052 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-home" (OuterVolumeSpecName: "home") pod "a037ac65-85de-46ce-8866-014de42ed204" (UID: "a037ac65-85de-46ce-8866-014de42ed204"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:26:36.424179 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.424074 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-model-cache" (OuterVolumeSpecName: "model-cache") pod "a037ac65-85de-46ce-8866-014de42ed204" (UID: "a037ac65-85de-46ce-8866-014de42ed204"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:26:36.424413 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.424367 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-tmp-dir\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:26:36.424459 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.424421 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-home\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:26:36.424459 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.424436 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-model-cache\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:26:36.426180 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.426157 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-dshm" (OuterVolumeSpecName: "dshm") pod "a037ac65-85de-46ce-8866-014de42ed204" (UID: "a037ac65-85de-46ce-8866-014de42ed204"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:26:36.426291 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.426157 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a037ac65-85de-46ce-8866-014de42ed204-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a037ac65-85de-46ce-8866-014de42ed204" (UID: "a037ac65-85de-46ce-8866-014de42ed204"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:26:36.426291 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.426217 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a037ac65-85de-46ce-8866-014de42ed204-kube-api-access-x4wpq" (OuterVolumeSpecName: "kube-api-access-x4wpq") pod "a037ac65-85de-46ce-8866-014de42ed204" (UID: "a037ac65-85de-46ce-8866-014de42ed204"). InnerVolumeSpecName "kube-api-access-x4wpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:26:36.525581 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.525558 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a037ac65-85de-46ce-8866-014de42ed204-dshm\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:26:36.525581 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.525580 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4wpq\" (UniqueName: \"kubernetes.io/projected/a037ac65-85de-46ce-8866-014de42ed204-kube-api-access-x4wpq\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:26:36.525728 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.525591 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a037ac65-85de-46ce-8866-014de42ed204-tls-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:26:36.726660 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.726573 2580 generic.go:356] "Generic (PLEG): container finished" podID="a037ac65-85de-46ce-8866-014de42ed204" containerID="eceb6778612ddd87126c5a40e55b573fb49c59cebf4dd6f6be8a656523e7536a" exitCode=0 Jul 21 22:26:36.726660 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.726616 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" Jul 21 22:26:36.726660 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.726630 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" event={"ID":"a037ac65-85de-46ce-8866-014de42ed204","Type":"ContainerDied","Data":"eceb6778612ddd87126c5a40e55b573fb49c59cebf4dd6f6be8a656523e7536a"} Jul 21 22:26:36.726875 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.726669 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9" event={"ID":"a037ac65-85de-46ce-8866-014de42ed204","Type":"ContainerDied","Data":"136c30c9dca8db6c55a858bf9f9ddd64398184c8f8440460b333256c3e1c5bf6"} Jul 21 22:26:36.726875 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.726689 2580 scope.go:117] "RemoveContainer" containerID="eceb6778612ddd87126c5a40e55b573fb49c59cebf4dd6f6be8a656523e7536a" Jul 21 22:26:36.740350 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.740331 2580 scope.go:117] "RemoveContainer" containerID="eceb6778612ddd87126c5a40e55b573fb49c59cebf4dd6f6be8a656523e7536a" Jul 21 22:26:36.740654 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:26:36.740634 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eceb6778612ddd87126c5a40e55b573fb49c59cebf4dd6f6be8a656523e7536a\": container with ID starting with eceb6778612ddd87126c5a40e55b573fb49c59cebf4dd6f6be8a656523e7536a not found: ID does not exist" containerID="eceb6778612ddd87126c5a40e55b573fb49c59cebf4dd6f6be8a656523e7536a" Jul 21 22:26:36.740700 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.740665 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eceb6778612ddd87126c5a40e55b573fb49c59cebf4dd6f6be8a656523e7536a"} err="failed to get container status \"eceb6778612ddd87126c5a40e55b573fb49c59cebf4dd6f6be8a656523e7536a\": rpc error: code = NotFound desc = could not find container \"eceb6778612ddd87126c5a40e55b573fb49c59cebf4dd6f6be8a656523e7536a\": container with ID starting with eceb6778612ddd87126c5a40e55b573fb49c59cebf4dd6f6be8a656523e7536a not found: ID does not exist" Jul 21 22:26:36.748450 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.748430 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9"] Jul 21 22:26:36.756541 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.756521 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4ad703/test-model-4ad703-kserve-56bb5c76d7-jqfv9"] Jul 21 22:26:36.779406 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:36.779359 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a037ac65-85de-46ce-8866-014de42ed204" path="/var/lib/kubelet/pods/a037ac65-85de-46ce-8866-014de42ed204/volumes" Jul 21 22:26:42.908560 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:42.908531 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9"] Jul 21 22:26:42.908965 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:42.908741 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" podUID="fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" containerName="istio-proxy" containerID="cri-o://d7cc60f62776718e6b570c5a2e9e2318df377a9c9cf955a06f94f7a80ff4c589" gracePeriod=30 Jul 21 22:26:48.167577 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.167520 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:26:48.244314 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.244286 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-workload-certs\") pod \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " Jul 21 22:26:48.244481 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.244333 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-podinfo\") pod \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " Jul 21 22:26:48.244481 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.244355 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpdtg\" (UniqueName: \"kubernetes.io/projected/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-kube-api-access-zpdtg\") pod \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " Jul 21 22:26:48.244481 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.244452 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-envoy\") pod \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " Jul 21 22:26:48.244631 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.244613 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-data\") pod \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " Jul 21 22:26:48.244631 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.244616 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" (UID: "fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:26:48.244720 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.244668 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istiod-ca-cert\") pod \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " Jul 21 22:26:48.244720 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.244695 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-credential-socket\") pod \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " Jul 21 22:26:48.244827 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.244754 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-token\") pod \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " Jul 21 22:26:48.244887 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.244854 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-workload-socket\") pod \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\" (UID: \"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a\") " Jul 21 22:26:48.245071 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.245021 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" (UID: "fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:26:48.245071 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.245057 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" (UID: "fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:26:48.245234 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.245214 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" (UID: "fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:26:48.245331 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.245314 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-workload-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:26:48.245408 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.245337 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-workload-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:26:48.245408 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.245351 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istiod-ca-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:26:48.245408 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.245363 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-credential-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:26:48.245537 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.245435 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-data" (OuterVolumeSpecName: "istio-data") pod "fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" (UID: "fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:26:48.247000 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.246968 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-token" (OuterVolumeSpecName: "istio-token") pod "fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" (UID: "fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:26:48.247107 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.247002 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" (UID: "fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 22:26:48.247107 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.247055 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" (UID: "fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:26:48.247107 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.247055 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-kube-api-access-zpdtg" (OuterVolumeSpecName: "kube-api-access-zpdtg") pod "fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" (UID: "fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a"). InnerVolumeSpecName "kube-api-access-zpdtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:26:48.346861 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.346824 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-token\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:26:48.346861 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.346854 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-podinfo\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:26:48.346861 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.346867 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpdtg\" (UniqueName: \"kubernetes.io/projected/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-kube-api-access-zpdtg\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:26:48.347103 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.346881 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-envoy\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:26:48.347103 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.346892 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a-istio-data\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:26:48.799932 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.799903 2580 generic.go:356] "Generic (PLEG): container finished" podID="fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" containerID="d7cc60f62776718e6b570c5a2e9e2318df377a9c9cf955a06f94f7a80ff4c589" exitCode=0 Jul 21 22:26:48.800080 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.799950 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" Jul 21 22:26:48.800080 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.799988 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" event={"ID":"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a","Type":"ContainerDied","Data":"d7cc60f62776718e6b570c5a2e9e2318df377a9c9cf955a06f94f7a80ff4c589"} Jul 21 22:26:48.800080 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.800041 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9" event={"ID":"fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a","Type":"ContainerDied","Data":"35a105ad6efa5dc20ddf4ce7681e1e1ccfdffb1810966c4092db4b68b93e93d3"} Jul 21 22:26:48.800080 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.800066 2580 scope.go:117] "RemoveContainer" containerID="d7cc60f62776718e6b570c5a2e9e2318df377a9c9cf955a06f94f7a80ff4c589" Jul 21 22:26:48.809264 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.809249 2580 scope.go:117] "RemoveContainer" containerID="cc9f0d8e1b588a568ea7663e4c6aceccba1111d37960970e2e0711a6639f1d64" Jul 21 22:26:48.817423 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.817348 2580 scope.go:117] "RemoveContainer" containerID="d7cc60f62776718e6b570c5a2e9e2318df377a9c9cf955a06f94f7a80ff4c589" Jul 21 22:26:48.817658 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:26:48.817638 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7cc60f62776718e6b570c5a2e9e2318df377a9c9cf955a06f94f7a80ff4c589\": container with ID starting with d7cc60f62776718e6b570c5a2e9e2318df377a9c9cf955a06f94f7a80ff4c589 not found: ID does not exist" containerID="d7cc60f62776718e6b570c5a2e9e2318df377a9c9cf955a06f94f7a80ff4c589" Jul 21 22:26:48.817755 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.817668 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7cc60f62776718e6b570c5a2e9e2318df377a9c9cf955a06f94f7a80ff4c589"} err="failed to get container status \"d7cc60f62776718e6b570c5a2e9e2318df377a9c9cf955a06f94f7a80ff4c589\": rpc error: code = NotFound desc = could not find container \"d7cc60f62776718e6b570c5a2e9e2318df377a9c9cf955a06f94f7a80ff4c589\": container with ID starting with d7cc60f62776718e6b570c5a2e9e2318df377a9c9cf955a06f94f7a80ff4c589 not found: ID does not exist" Jul 21 22:26:48.817755 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.817683 2580 scope.go:117] "RemoveContainer" containerID="cc9f0d8e1b588a568ea7663e4c6aceccba1111d37960970e2e0711a6639f1d64" Jul 21 22:26:48.817988 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:26:48.817967 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc9f0d8e1b588a568ea7663e4c6aceccba1111d37960970e2e0711a6639f1d64\": container with ID starting with cc9f0d8e1b588a568ea7663e4c6aceccba1111d37960970e2e0711a6639f1d64 not found: ID does not exist" containerID="cc9f0d8e1b588a568ea7663e4c6aceccba1111d37960970e2e0711a6639f1d64" Jul 21 22:26:48.818040 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.817996 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc9f0d8e1b588a568ea7663e4c6aceccba1111d37960970e2e0711a6639f1d64"} err="failed to get container status \"cc9f0d8e1b588a568ea7663e4c6aceccba1111d37960970e2e0711a6639f1d64\": rpc error: code = NotFound desc = could not find container \"cc9f0d8e1b588a568ea7663e4c6aceccba1111d37960970e2e0711a6639f1d64\": container with ID starting with cc9f0d8e1b588a568ea7663e4c6aceccba1111d37960970e2e0711a6639f1d64 not found: ID does not exist" Jul 21 22:26:48.819710 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.819692 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9"] Jul 21 22:26:48.823998 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:48.823980 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3f5ae6-openshift-default-7bc7b6fdd5-z7rp9"] Jul 21 22:26:50.779718 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:26:50.779686 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" path="/var/lib/kubelet/pods/fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a/volumes" Jul 21 22:27:04.900516 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:04.900485 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j"] Jul 21 22:27:04.901141 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:04.900731 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" podUID="1ba33d73-a499-4d7f-abe9-fce4ccfec92e" containerName="istio-proxy" containerID="cri-o://aee4f00dfcc0d2b11e16da1a82e2b54d59a8f5add8c828767356bddca20e388f" gracePeriod=30 Jul 21 22:27:05.800548 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.800515 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf"] Jul 21 22:27:05.801245 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.801209 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" containerName="istio-proxy" Jul 21 22:27:05.801245 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.801236 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" containerName="istio-proxy" Jul 21 22:27:05.801245 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.801250 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a037ac65-85de-46ce-8866-014de42ed204" containerName="main" Jul 21 22:27:05.801512 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.801256 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a037ac65-85de-46ce-8866-014de42ed204" containerName="main" Jul 21 22:27:05.801512 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.801347 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" containerName="istio-proxy" Jul 21 22:27:05.801512 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.801356 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a037ac65-85de-46ce-8866-014de42ed204" containerName="main" Jul 21 22:27:05.804790 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.804771 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:05.807208 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.807188 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-09bf16-openshift-default-dockercfg-rstdm\"" Jul 21 22:27:05.817560 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.817525 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf"] Jul 21 22:27:05.916145 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.916108 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-workload-socket\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:05.916741 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.916711 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-token\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:05.916883 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.916857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-workload-certs\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:05.916966 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.916902 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vztm7\" (UniqueName: \"kubernetes.io/projected/303027a2-5fed-4fc4-a554-29a28e88f2f6-kube-api-access-vztm7\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:05.916966 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.916939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-podinfo\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:05.917085 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.917006 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-credential-socket\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:05.917085 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.917042 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-envoy\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:05.917085 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.917071 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303027a2-5fed-4fc4-a554-29a28e88f2f6-istiod-ca-cert\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:05.917259 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:05.917105 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-data\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.018585 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.018554 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-data\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.018768 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.018619 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-workload-socket\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.018768 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.018643 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-token\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.018768 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.018685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-workload-certs\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.018768 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.018703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vztm7\" (UniqueName: \"kubernetes.io/projected/303027a2-5fed-4fc4-a554-29a28e88f2f6-kube-api-access-vztm7\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.018768 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.018722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-podinfo\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.018768 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.018763 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-credential-socket\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.019093 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.018787 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-envoy\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.019093 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.018812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303027a2-5fed-4fc4-a554-29a28e88f2f6-istiod-ca-cert\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.019093 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.019012 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-data\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.019093 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.019036 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-workload-socket\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.019289 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.019101 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-workload-certs\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.019289 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.019197 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-credential-socket\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.019629 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.019608 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303027a2-5fed-4fc4-a554-29a28e88f2f6-istiod-ca-cert\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.021202 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.021181 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-envoy\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.021488 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.021467 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-podinfo\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.026699 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.026673 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-token\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.027173 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.027154 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vztm7\" (UniqueName: \"kubernetes.io/projected/303027a2-5fed-4fc4-a554-29a28e88f2f6-kube-api-access-vztm7\") pod \"e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.116852 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.116749 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:06.258786 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.258757 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf"] Jul 21 22:27:06.261208 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:27:06.261179 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303027a2_5fed_4fc4_a554_29a28e88f2f6.slice/crio-b58347d20007cf772cc31c960c381a215c93872d010a47ee304165b8740cf5e4 WatchSource:0}: Error finding container b58347d20007cf772cc31c960c381a215c93872d010a47ee304165b8740cf5e4: Status 404 returned error can't find the container with id b58347d20007cf772cc31c960c381a215c93872d010a47ee304165b8740cf5e4 Jul 21 22:27:06.263649 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.263617 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:27:06.263737 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.263689 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:27:06.263737 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.263725 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:27:06.896536 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.896495 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" event={"ID":"303027a2-5fed-4fc4-a554-29a28e88f2f6","Type":"ContainerStarted","Data":"aa91e585538e8b5165716cc673e4a1b66f7e8c3ea39624e397f46df4a54c8938"} Jul 21 22:27:06.896698 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.896550 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" event={"ID":"303027a2-5fed-4fc4-a554-29a28e88f2f6","Type":"ContainerStarted","Data":"b58347d20007cf772cc31c960c381a215c93872d010a47ee304165b8740cf5e4"} Jul 21 22:27:06.919661 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:06.919610 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" podStartSLOduration=1.919594276 podStartE2EDuration="1.919594276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:27:06.917443511 +0000 UTC m=+2304.680677771" watchObservedRunningTime="2026-07-21 22:27:06.919594276 +0000 UTC m=+2304.682828538" Jul 21 22:27:07.117442 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:07.117401 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:07.122408 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:07.122365 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:07.902899 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:07.902869 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:07.903996 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:07.903977 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:10.160993 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.160933 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:27:10.261259 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.261227 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-workload-socket\") pod \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " Jul 21 22:27:10.261431 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.261281 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-data\") pod \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " Jul 21 22:27:10.261431 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.261307 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istiod-ca-cert\") pod \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " Jul 21 22:27:10.261431 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.261328 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh7s4\" (UniqueName: \"kubernetes.io/projected/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-kube-api-access-lh7s4\") pod \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " Jul 21 22:27:10.261431 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.261383 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-podinfo\") pod \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " Jul 21 22:27:10.261431 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.261412 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-envoy\") pod \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " Jul 21 22:27:10.261710 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.261435 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-workload-certs\") pod \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " Jul 21 22:27:10.261710 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.261511 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-token\") pod \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " Jul 21 22:27:10.261710 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.261558 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ba33d73-a499-4d7f-abe9-fce4ccfec92e" (UID: "1ba33d73-a499-4d7f-abe9-fce4ccfec92e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:27:10.261710 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.261599 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-credential-socket\") pod \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\" (UID: \"1ba33d73-a499-4d7f-abe9-fce4ccfec92e\") " Jul 21 22:27:10.261923 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.261744 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ba33d73-a499-4d7f-abe9-fce4ccfec92e" (UID: "1ba33d73-a499-4d7f-abe9-fce4ccfec92e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:27:10.261923 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.261758 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ba33d73-a499-4d7f-abe9-fce4ccfec92e" (UID: "1ba33d73-a499-4d7f-abe9-fce4ccfec92e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:27:10.262034 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.261943 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ba33d73-a499-4d7f-abe9-fce4ccfec92e" (UID: "1ba33d73-a499-4d7f-abe9-fce4ccfec92e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:27:10.262034 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.261991 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-workload-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:27:10.262034 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.262011 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istiod-ca-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:27:10.262034 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.262026 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-workload-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:27:10.262550 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.262455 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ba33d73-a499-4d7f-abe9-fce4ccfec92e" (UID: "1ba33d73-a499-4d7f-abe9-fce4ccfec92e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:27:10.263828 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.263801 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-kube-api-access-lh7s4" (OuterVolumeSpecName: "kube-api-access-lh7s4") pod "1ba33d73-a499-4d7f-abe9-fce4ccfec92e" (UID: "1ba33d73-a499-4d7f-abe9-fce4ccfec92e"). InnerVolumeSpecName "kube-api-access-lh7s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:27:10.263828 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.263813 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ba33d73-a499-4d7f-abe9-fce4ccfec92e" (UID: "1ba33d73-a499-4d7f-abe9-fce4ccfec92e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:27:10.263970 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.263845 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ba33d73-a499-4d7f-abe9-fce4ccfec92e" (UID: "1ba33d73-a499-4d7f-abe9-fce4ccfec92e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 22:27:10.264060 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.264032 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ba33d73-a499-4d7f-abe9-fce4ccfec92e" (UID: "1ba33d73-a499-4d7f-abe9-fce4ccfec92e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:27:10.362878 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.362843 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-credential-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:27:10.362878 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.362872 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-data\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:27:10.362878 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.362882 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh7s4\" (UniqueName: \"kubernetes.io/projected/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-kube-api-access-lh7s4\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:27:10.363122 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.362893 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-podinfo\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:27:10.363122 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.362902 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-envoy\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:27:10.363122 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.362910 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba33d73-a499-4d7f-abe9-fce4ccfec92e-istio-token\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:27:10.922233 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.922200 2580 generic.go:356] "Generic (PLEG): container finished" podID="1ba33d73-a499-4d7f-abe9-fce4ccfec92e" containerID="aee4f00dfcc0d2b11e16da1a82e2b54d59a8f5add8c828767356bddca20e388f" exitCode=0 Jul 21 22:27:10.922438 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.922252 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" Jul 21 22:27:10.922438 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.922295 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" event={"ID":"1ba33d73-a499-4d7f-abe9-fce4ccfec92e","Type":"ContainerDied","Data":"aee4f00dfcc0d2b11e16da1a82e2b54d59a8f5add8c828767356bddca20e388f"} Jul 21 22:27:10.922438 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.922353 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j" event={"ID":"1ba33d73-a499-4d7f-abe9-fce4ccfec92e","Type":"ContainerDied","Data":"2b29aaada8d0929cb24987d958ccb7873df1b57e2396441c1fa73f234e3d1e98"} Jul 21 22:27:10.922438 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.922401 2580 scope.go:117] "RemoveContainer" containerID="aee4f00dfcc0d2b11e16da1a82e2b54d59a8f5add8c828767356bddca20e388f" Jul 21 22:27:10.937134 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.937105 2580 scope.go:117] "RemoveContainer" containerID="18fb678378f084d4bf3dbfb4be649c015a56755a7892dccd27e78ab1c0c57713" Jul 21 22:27:10.942492 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.942462 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j"] Jul 21 22:27:10.946244 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.946222 2580 scope.go:117] "RemoveContainer" containerID="aee4f00dfcc0d2b11e16da1a82e2b54d59a8f5add8c828767356bddca20e388f" Jul 21 22:27:10.946551 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:27:10.946530 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aee4f00dfcc0d2b11e16da1a82e2b54d59a8f5add8c828767356bddca20e388f\": container with ID starting with aee4f00dfcc0d2b11e16da1a82e2b54d59a8f5add8c828767356bddca20e388f not found: ID does not exist" containerID="aee4f00dfcc0d2b11e16da1a82e2b54d59a8f5add8c828767356bddca20e388f" Jul 21 22:27:10.946652 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.946563 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee4f00dfcc0d2b11e16da1a82e2b54d59a8f5add8c828767356bddca20e388f"} err="failed to get container status \"aee4f00dfcc0d2b11e16da1a82e2b54d59a8f5add8c828767356bddca20e388f\": rpc error: code = NotFound desc = could not find container \"aee4f00dfcc0d2b11e16da1a82e2b54d59a8f5add8c828767356bddca20e388f\": container with ID starting with aee4f00dfcc0d2b11e16da1a82e2b54d59a8f5add8c828767356bddca20e388f not found: ID does not exist" Jul 21 22:27:10.946652 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.946586 2580 scope.go:117] "RemoveContainer" containerID="18fb678378f084d4bf3dbfb4be649c015a56755a7892dccd27e78ab1c0c57713" Jul 21 22:27:10.946854 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:27:10.946836 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18fb678378f084d4bf3dbfb4be649c015a56755a7892dccd27e78ab1c0c57713\": container with ID starting with 18fb678378f084d4bf3dbfb4be649c015a56755a7892dccd27e78ab1c0c57713 not found: ID does not exist" containerID="18fb678378f084d4bf3dbfb4be649c015a56755a7892dccd27e78ab1c0c57713" Jul 21 22:27:10.946889 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.946861 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18fb678378f084d4bf3dbfb4be649c015a56755a7892dccd27e78ab1c0c57713"} err="failed to get container status \"18fb678378f084d4bf3dbfb4be649c015a56755a7892dccd27e78ab1c0c57713\": rpc error: code = NotFound desc = could not find container \"18fb678378f084d4bf3dbfb4be649c015a56755a7892dccd27e78ab1c0c57713\": container with ID starting with 18fb678378f084d4bf3dbfb4be649c015a56755a7892dccd27e78ab1c0c57713 not found: ID does not exist" Jul 21 22:27:10.946930 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:10.946899 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4ad703-openshift-default-87d74fc-p274j"] Jul 21 22:27:12.779536 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:12.779505 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ba33d73-a499-4d7f-abe9-fce4ccfec92e" path="/var/lib/kubelet/pods/1ba33d73-a499-4d7f-abe9-fce4ccfec92e/volumes" Jul 21 22:27:14.107350 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.107309 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z"] Jul 21 22:27:14.107874 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.107855 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" containerName="istio-proxy" Jul 21 22:27:14.107874 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.107874 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" containerName="istio-proxy" Jul 21 22:27:14.108016 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.107893 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ba33d73-a499-4d7f-abe9-fce4ccfec92e" containerName="istio-proxy" Jul 21 22:27:14.108016 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.107899 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ba33d73-a499-4d7f-abe9-fce4ccfec92e" containerName="istio-proxy" Jul 21 22:27:14.108016 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.107907 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ba33d73-a499-4d7f-abe9-fce4ccfec92e" containerName="istio-proxy" Jul 21 22:27:14.108016 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.107912 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ba33d73-a499-4d7f-abe9-fce4ccfec92e" containerName="istio-proxy" Jul 21 22:27:14.108016 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.107998 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ba33d73-a499-4d7f-abe9-fce4ccfec92e" containerName="istio-proxy" Jul 21 22:27:14.108016 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.108009 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ba33d73-a499-4d7f-abe9-fce4ccfec92e" containerName="istio-proxy" Jul 21 22:27:14.108016 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.108019 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fab9a916-e8e9-4ce8-a9d1-a93e4abfe62a" containerName="istio-proxy" Jul 21 22:27:14.112608 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.112591 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.115970 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.115947 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-76e244-openshift-default-dockercfg-8p6v4\"" Jul 21 22:27:14.123561 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.123536 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z"] Jul 21 22:27:14.199554 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.199511 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-workload-certs\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.199777 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.199584 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-credential-socket\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.199777 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.199690 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-workload-socket\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.199777 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.199770 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-token\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.200016 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.199803 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-data\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.200016 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.199846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-envoy\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.200016 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.199881 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx2wt\" (UniqueName: \"kubernetes.io/projected/d4ecf4cf-1065-4529-9612-9b1e9d527204-kube-api-access-jx2wt\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.200016 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.199941 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-podinfo\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.200016 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.199969 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4ecf4cf-1065-4529-9612-9b1e9d527204-istiod-ca-cert\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.301585 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.301544 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-token\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.301585 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.301582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-data\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.301840 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.301606 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-envoy\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.301840 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.301625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx2wt\" (UniqueName: \"kubernetes.io/projected/d4ecf4cf-1065-4529-9612-9b1e9d527204-kube-api-access-jx2wt\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.301840 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.301661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-podinfo\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.301840 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.301678 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4ecf4cf-1065-4529-9612-9b1e9d527204-istiod-ca-cert\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.301840 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.301737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-workload-certs\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.301840 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.301780 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-credential-socket\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.301840 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.301810 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-workload-socket\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.302195 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.302071 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-data\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.302195 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.302165 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-workload-certs\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.302195 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.302190 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-credential-socket\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.302305 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.302242 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-workload-socket\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.302534 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.302517 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4ecf4cf-1065-4529-9612-9b1e9d527204-istiod-ca-cert\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.304654 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.304634 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-envoy\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.304896 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.304875 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-podinfo\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.309801 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.309780 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx2wt\" (UniqueName: \"kubernetes.io/projected/d4ecf4cf-1065-4529-9612-9b1e9d527204-kube-api-access-jx2wt\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.309882 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.309850 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-token\") pod \"e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.424121 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.424031 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:14.568283 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.568253 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z"] Jul 21 22:27:14.569916 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:27:14.569892 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ecf4cf_1065_4529_9612_9b1e9d527204.slice/crio-fbbbd2613dcd6353539ac34f4040d5fa48b8b8840f4df4be74f560f81c690d23 WatchSource:0}: Error finding container fbbbd2613dcd6353539ac34f4040d5fa48b8b8840f4df4be74f560f81c690d23: Status 404 returned error can't find the container with id fbbbd2613dcd6353539ac34f4040d5fa48b8b8840f4df4be74f560f81c690d23 Jul 21 22:27:14.572133 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.572102 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:27:14.572203 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.572170 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:27:14.572203 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.572198 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:27:14.944677 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.944636 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" event={"ID":"d4ecf4cf-1065-4529-9612-9b1e9d527204","Type":"ContainerStarted","Data":"09659ad7aedef76307c2f10f81626e86ce971e9b29b0775f4806afb06256aa5f"} Jul 21 22:27:14.944870 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.944699 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" event={"ID":"d4ecf4cf-1065-4529-9612-9b1e9d527204","Type":"ContainerStarted","Data":"fbbbd2613dcd6353539ac34f4040d5fa48b8b8840f4df4be74f560f81c690d23"} Jul 21 22:27:14.968934 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:14.968864 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" podStartSLOduration=0.968841063 podStartE2EDuration="968.841063ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:27:14.965492002 +0000 UTC m=+2312.728726289" watchObservedRunningTime="2026-07-21 22:27:14.968841063 +0000 UTC m=+2312.732075325" Jul 21 22:27:15.425121 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:15.425083 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:15.430301 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:15.430275 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:15.950470 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:15.950442 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:15.951623 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:15.951600 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:23.762184 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:23.761294 2580 patch_prober.go:28] interesting pod/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": read tcp 10.134.0.2:60224->10.134.0.61:15021: read: connection reset by peer" start-of-body= Jul 21 22:27:23.762184 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:23.761418 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" podUID="303027a2-5fed-4fc4-a554-29a28e88f2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": read tcp 10.134.0.2:60224->10.134.0.61:15021: read: connection reset by peer" Jul 21 22:27:24.016929 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:24.016851 2580 generic.go:356] "Generic (PLEG): container finished" podID="303027a2-5fed-4fc4-a554-29a28e88f2f6" containerID="aa91e585538e8b5165716cc673e4a1b66f7e8c3ea39624e397f46df4a54c8938" exitCode=0 Jul 21 22:27:24.017072 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:24.016927 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" event={"ID":"303027a2-5fed-4fc4-a554-29a28e88f2f6","Type":"ContainerDied","Data":"aa91e585538e8b5165716cc673e4a1b66f7e8c3ea39624e397f46df4a54c8938"} Jul 21 22:27:24.017365 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:24.017339 2580 scope.go:117] "RemoveContainer" containerID="aa91e585538e8b5165716cc673e4a1b66f7e8c3ea39624e397f46df4a54c8938" Jul 21 22:27:24.018329 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:24.018301 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:27:24.018403 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:24.018388 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:27:24.018454 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:24.018422 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:27:25.024049 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:25.024011 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" event={"ID":"303027a2-5fed-4fc4-a554-29a28e88f2f6","Type":"ContainerStarted","Data":"4e238c78df8181b0738a0b915b82196c0cc3a9ae47f7272842fdbdc5afc57717"} Jul 21 22:27:25.117062 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:25.117021 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:25.117334 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:25.117311 2580 patch_prober.go:28] interesting pod/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 21 22:27:25.117420 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:25.117402 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" podUID="303027a2-5fed-4fc4-a554-29a28e88f2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 21 22:27:26.118139 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:26.118094 2580 patch_prober.go:28] interesting pod/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 21 22:27:26.118817 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:26.118776 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" podUID="303027a2-5fed-4fc4-a554-29a28e88f2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 21 22:27:27.118145 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:27.118112 2580 patch_prober.go:28] interesting pod/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 21 22:27:27.118615 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:27.118497 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" podUID="303027a2-5fed-4fc4-a554-29a28e88f2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 21 22:27:28.117428 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:28.117359 2580 patch_prober.go:28] interesting pod/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 21 22:27:28.117735 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:28.117707 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" podUID="303027a2-5fed-4fc4-a554-29a28e88f2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 21 22:27:29.121025 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:29.120980 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:29.121432 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:29.121327 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:29.121863 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:29.121844 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:27:31.992572 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:31.992533 2580 patch_prober.go:28] interesting pod/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 21 22:27:31.992973 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:31.992603 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" podUID="d4ecf4cf-1065-4529-9612-9b1e9d527204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 21 22:27:34.080345 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:34.080309 2580 generic.go:356] "Generic (PLEG): container finished" podID="d4ecf4cf-1065-4529-9612-9b1e9d527204" containerID="09659ad7aedef76307c2f10f81626e86ce971e9b29b0775f4806afb06256aa5f" exitCode=0 Jul 21 22:27:34.080811 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:34.080394 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" event={"ID":"d4ecf4cf-1065-4529-9612-9b1e9d527204","Type":"ContainerDied","Data":"09659ad7aedef76307c2f10f81626e86ce971e9b29b0775f4806afb06256aa5f"} Jul 21 22:27:34.080860 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:34.080834 2580 scope.go:117] "RemoveContainer" containerID="09659ad7aedef76307c2f10f81626e86ce971e9b29b0775f4806afb06256aa5f" Jul 21 22:27:34.081869 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:34.081839 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:27:34.081963 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:34.081913 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:27:34.082005 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:34.081971 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:27:35.093149 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:35.093112 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" event={"ID":"d4ecf4cf-1065-4529-9612-9b1e9d527204","Type":"ContainerStarted","Data":"9d75edfc02d50994d3fabe73c8a4d3df523a9f28666bc36b417b4f10a5f8c1d4"} Jul 21 22:27:35.425222 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:35.425131 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:35.425426 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:35.425403 2580 patch_prober.go:28] interesting pod/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 21 22:27:35.425507 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:35.425486 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" podUID="d4ecf4cf-1065-4529-9612-9b1e9d527204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 21 22:27:36.424754 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:36.424712 2580 patch_prober.go:28] interesting pod/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 21 22:27:36.425235 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:36.424775 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" podUID="d4ecf4cf-1065-4529-9612-9b1e9d527204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 21 22:27:37.424424 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:37.424365 2580 patch_prober.go:28] interesting pod/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 21 22:27:37.424627 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:37.424449 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" podUID="d4ecf4cf-1065-4529-9612-9b1e9d527204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 21 22:27:38.424531 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:38.424488 2580 patch_prober.go:28] interesting pod/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 21 22:27:38.425064 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:38.424548 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" podUID="d4ecf4cf-1065-4529-9612-9b1e9d527204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 21 22:27:39.429093 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:39.429063 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:39.429518 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:39.429497 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:39.430329 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:39.430308 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:27:42.960591 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:42.960555 2580 scope.go:117] "RemoveContainer" containerID="0f8384c7ab273311768c02ce1de6ba750411409aab34be9b11cda7d18948e7a2" Jul 21 22:27:56.245815 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:56.245779 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf"] Jul 21 22:27:56.246293 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:56.246092 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" podUID="303027a2-5fed-4fc4-a554-29a28e88f2f6" containerName="istio-proxy" containerID="cri-o://4e238c78df8181b0738a0b915b82196c0cc3a9ae47f7272842fdbdc5afc57717" gracePeriod=30 Jul 21 22:27:59.123921 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:59.123882 2580 patch_prober.go:28] interesting pod/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 22:27:59.124293 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:27:59.123941 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" podUID="303027a2-5fed-4fc4-a554-29a28e88f2f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 22:28:01.508040 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.508009 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:28:01.596623 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.596537 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-workload-certs\") pod \"303027a2-5fed-4fc4-a554-29a28e88f2f6\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " Jul 21 22:28:01.596623 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.596579 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-envoy\") pod \"303027a2-5fed-4fc4-a554-29a28e88f2f6\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " Jul 21 22:28:01.596623 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.596605 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-credential-socket\") pod \"303027a2-5fed-4fc4-a554-29a28e88f2f6\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " Jul 21 22:28:01.596901 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.596701 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vztm7\" (UniqueName: \"kubernetes.io/projected/303027a2-5fed-4fc4-a554-29a28e88f2f6-kube-api-access-vztm7\") pod \"303027a2-5fed-4fc4-a554-29a28e88f2f6\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " Jul 21 22:28:01.596901 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.596725 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303027a2-5fed-4fc4-a554-29a28e88f2f6-istiod-ca-cert\") pod \"303027a2-5fed-4fc4-a554-29a28e88f2f6\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " Jul 21 22:28:01.596901 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.596750 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-podinfo\") pod \"303027a2-5fed-4fc4-a554-29a28e88f2f6\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " Jul 21 22:28:01.596901 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.596807 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-data\") pod \"303027a2-5fed-4fc4-a554-29a28e88f2f6\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " Jul 21 22:28:01.596901 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.596855 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-token\") pod \"303027a2-5fed-4fc4-a554-29a28e88f2f6\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " Jul 21 22:28:01.596901 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.596892 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-workload-socket\") pod \"303027a2-5fed-4fc4-a554-29a28e88f2f6\" (UID: \"303027a2-5fed-4fc4-a554-29a28e88f2f6\") " Jul 21 22:28:01.597190 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.596933 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "303027a2-5fed-4fc4-a554-29a28e88f2f6" (UID: "303027a2-5fed-4fc4-a554-29a28e88f2f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:28:01.597190 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.596963 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "303027a2-5fed-4fc4-a554-29a28e88f2f6" (UID: "303027a2-5fed-4fc4-a554-29a28e88f2f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:28:01.597324 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.597252 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "303027a2-5fed-4fc4-a554-29a28e88f2f6" (UID: "303027a2-5fed-4fc4-a554-29a28e88f2f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:28:01.597496 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.597283 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-workload-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:28:01.597545 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.597511 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-credential-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:28:01.597545 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.597503 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/303027a2-5fed-4fc4-a554-29a28e88f2f6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "303027a2-5fed-4fc4-a554-29a28e88f2f6" (UID: "303027a2-5fed-4fc4-a554-29a28e88f2f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:28:01.598220 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.598186 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-data" (OuterVolumeSpecName: "istio-data") pod "303027a2-5fed-4fc4-a554-29a28e88f2f6" (UID: "303027a2-5fed-4fc4-a554-29a28e88f2f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:28:01.598904 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.598877 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "303027a2-5fed-4fc4-a554-29a28e88f2f6" (UID: "303027a2-5fed-4fc4-a554-29a28e88f2f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:28:01.599077 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.599059 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "303027a2-5fed-4fc4-a554-29a28e88f2f6" (UID: "303027a2-5fed-4fc4-a554-29a28e88f2f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 22:28:01.599502 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.599486 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303027a2-5fed-4fc4-a554-29a28e88f2f6-kube-api-access-vztm7" (OuterVolumeSpecName: "kube-api-access-vztm7") pod "303027a2-5fed-4fc4-a554-29a28e88f2f6" (UID: "303027a2-5fed-4fc4-a554-29a28e88f2f6"). InnerVolumeSpecName "kube-api-access-vztm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:28:01.599842 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.599827 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-token" (OuterVolumeSpecName: "istio-token") pod "303027a2-5fed-4fc4-a554-29a28e88f2f6" (UID: "303027a2-5fed-4fc4-a554-29a28e88f2f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:28:01.698068 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.698030 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-envoy\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:28:01.698243 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.698084 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vztm7\" (UniqueName: \"kubernetes.io/projected/303027a2-5fed-4fc4-a554-29a28e88f2f6-kube-api-access-vztm7\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:28:01.698243 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.698095 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303027a2-5fed-4fc4-a554-29a28e88f2f6-istiod-ca-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:28:01.698243 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.698103 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-podinfo\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:28:01.698243 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.698111 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-data\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:28:01.698243 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.698118 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303027a2-5fed-4fc4-a554-29a28e88f2f6-istio-token\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:28:01.698243 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:01.698126 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303027a2-5fed-4fc4-a554-29a28e88f2f6-workload-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:28:02.235556 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:02.235518 2580 generic.go:356] "Generic (PLEG): container finished" podID="303027a2-5fed-4fc4-a554-29a28e88f2f6" containerID="4e238c78df8181b0738a0b915b82196c0cc3a9ae47f7272842fdbdc5afc57717" exitCode=0 Jul 21 22:28:02.235773 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:02.235589 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" Jul 21 22:28:02.235773 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:02.235614 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" event={"ID":"303027a2-5fed-4fc4-a554-29a28e88f2f6","Type":"ContainerDied","Data":"4e238c78df8181b0738a0b915b82196c0cc3a9ae47f7272842fdbdc5afc57717"} Jul 21 22:28:02.235773 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:02.235694 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf" event={"ID":"303027a2-5fed-4fc4-a554-29a28e88f2f6","Type":"ContainerDied","Data":"b58347d20007cf772cc31c960c381a215c93872d010a47ee304165b8740cf5e4"} Jul 21 22:28:02.235773 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:02.235718 2580 scope.go:117] "RemoveContainer" containerID="4e238c78df8181b0738a0b915b82196c0cc3a9ae47f7272842fdbdc5afc57717" Jul 21 22:28:02.245667 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:02.245643 2580 scope.go:117] "RemoveContainer" containerID="aa91e585538e8b5165716cc673e4a1b66f7e8c3ea39624e397f46df4a54c8938" Jul 21 22:28:02.254769 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:02.254745 2580 scope.go:117] "RemoveContainer" containerID="4e238c78df8181b0738a0b915b82196c0cc3a9ae47f7272842fdbdc5afc57717" Jul 21 22:28:02.255048 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:28:02.255028 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e238c78df8181b0738a0b915b82196c0cc3a9ae47f7272842fdbdc5afc57717\": container with ID starting with 4e238c78df8181b0738a0b915b82196c0cc3a9ae47f7272842fdbdc5afc57717 not found: ID does not exist" containerID="4e238c78df8181b0738a0b915b82196c0cc3a9ae47f7272842fdbdc5afc57717" Jul 21 22:28:02.255183 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:02.255058 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e238c78df8181b0738a0b915b82196c0cc3a9ae47f7272842fdbdc5afc57717"} err="failed to get container status \"4e238c78df8181b0738a0b915b82196c0cc3a9ae47f7272842fdbdc5afc57717\": rpc error: code = NotFound desc = could not find container \"4e238c78df8181b0738a0b915b82196c0cc3a9ae47f7272842fdbdc5afc57717\": container with ID starting with 4e238c78df8181b0738a0b915b82196c0cc3a9ae47f7272842fdbdc5afc57717 not found: ID does not exist" Jul 21 22:28:02.255183 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:02.255077 2580 scope.go:117] "RemoveContainer" containerID="aa91e585538e8b5165716cc673e4a1b66f7e8c3ea39624e397f46df4a54c8938" Jul 21 22:28:02.255340 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:28:02.255320 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa91e585538e8b5165716cc673e4a1b66f7e8c3ea39624e397f46df4a54c8938\": container with ID starting with aa91e585538e8b5165716cc673e4a1b66f7e8c3ea39624e397f46df4a54c8938 not found: ID does not exist" containerID="aa91e585538e8b5165716cc673e4a1b66f7e8c3ea39624e397f46df4a54c8938" Jul 21 22:28:02.255426 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:02.255346 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa91e585538e8b5165716cc673e4a1b66f7e8c3ea39624e397f46df4a54c8938"} err="failed to get container status \"aa91e585538e8b5165716cc673e4a1b66f7e8c3ea39624e397f46df4a54c8938\": rpc error: code = NotFound desc = could not find container \"aa91e585538e8b5165716cc673e4a1b66f7e8c3ea39624e397f46df4a54c8938\": container with ID starting with aa91e585538e8b5165716cc673e4a1b66f7e8c3ea39624e397f46df4a54c8938 not found: ID does not exist" Jul 21 22:28:02.262006 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:02.261976 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf"] Jul 21 22:28:02.265455 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:02.265429 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-09bf16-openshift-default-577d9f7555-z7dkf"] Jul 21 22:28:02.780521 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:02.780493 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="303027a2-5fed-4fc4-a554-29a28e88f2f6" path="/var/lib/kubelet/pods/303027a2-5fed-4fc4-a554-29a28e88f2f6/volumes" Jul 21 22:28:23.570966 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:23.570928 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z"] Jul 21 22:28:23.571605 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:23.571172 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" podUID="d4ecf4cf-1065-4529-9612-9b1e9d527204" containerName="istio-proxy" containerID="cri-o://9d75edfc02d50994d3fabe73c8a4d3df523a9f28666bc36b417b4f10a5f8c1d4" gracePeriod=30 Jul 21 22:28:24.432047 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.432007 2580 patch_prober.go:28] interesting pod/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 22:28:24.432221 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.432065 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" podUID="d4ecf4cf-1065-4529-9612-9b1e9d527204" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 22:28:24.524504 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.524469 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6"] Jul 21 22:28:24.524920 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.524908 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="303027a2-5fed-4fc4-a554-29a28e88f2f6" containerName="istio-proxy" Jul 21 22:28:24.524964 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.524922 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="303027a2-5fed-4fc4-a554-29a28e88f2f6" containerName="istio-proxy" Jul 21 22:28:24.525010 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.525001 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="303027a2-5fed-4fc4-a554-29a28e88f2f6" containerName="istio-proxy" Jul 21 22:28:24.525045 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.525010 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="303027a2-5fed-4fc4-a554-29a28e88f2f6" containerName="istio-proxy" Jul 21 22:28:24.528155 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.528134 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.531156 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.531130 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-131f31-openshift-default-dockercfg-w6j2j\"" Jul 21 22:28:24.561856 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.561833 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6"] Jul 21 22:28:24.635467 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.635417 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6vvg\" (UniqueName: \"kubernetes.io/projected/aeb92da7-67c6-449f-81a3-e5c53fd73e69-kube-api-access-p6vvg\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.635963 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.635526 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-data\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.635963 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.635570 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-envoy\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.635963 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.635614 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-token\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.635963 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.635665 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-workload-socket\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.635963 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.635690 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istiod-ca-cert\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.635963 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.635753 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-podinfo\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.635963 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.635859 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-workload-certs\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.635963 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.635954 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-credential-socket\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.737333 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.737305 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-workload-socket\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.737535 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.737337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istiod-ca-cert\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.737535 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.737356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-podinfo\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.737535 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.737439 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-workload-certs\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.737535 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.737493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-credential-socket\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.737535 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.737524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6vvg\" (UniqueName: \"kubernetes.io/projected/aeb92da7-67c6-449f-81a3-e5c53fd73e69-kube-api-access-p6vvg\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.737806 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.737567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-data\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.737806 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.737599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-envoy\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.737806 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.737628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-token\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.737806 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.737777 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-workload-socket\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.738020 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.737808 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-workload-certs\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.738020 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.737903 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-credential-socket\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.738020 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.737931 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-data\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.738219 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.738195 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istiod-ca-cert\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.740135 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.740116 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-envoy\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.740240 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.740224 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-podinfo\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.753808 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.753782 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-token\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.753924 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.753792 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6vvg\" (UniqueName: \"kubernetes.io/projected/aeb92da7-67c6-449f-81a3-e5c53fd73e69-kube-api-access-p6vvg\") pod \"e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.838316 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.838278 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:24.981537 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.981508 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6"] Jul 21 22:28:24.983032 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:28:24.983003 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb92da7_67c6_449f_81a3_e5c53fd73e69.slice/crio-bbd7f97be3a60c487b3b969b0b7a0f43e9af9c079bd53eb2147c3f61336197b6 WatchSource:0}: Error finding container bbd7f97be3a60c487b3b969b0b7a0f43e9af9c079bd53eb2147c3f61336197b6: Status 404 returned error can't find the container with id bbd7f97be3a60c487b3b969b0b7a0f43e9af9c079bd53eb2147c3f61336197b6 Jul 21 22:28:24.985176 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.985141 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:28:24.985248 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.985207 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:28:24.985248 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:24.985238 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:28:25.359575 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:25.359478 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" event={"ID":"aeb92da7-67c6-449f-81a3-e5c53fd73e69","Type":"ContainerStarted","Data":"8220a49ab3c748d49de60c14fcbefc48a8a7a80d4e9bc808a86f3d031710f07a"} Jul 21 22:28:25.359575 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:25.359536 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" event={"ID":"aeb92da7-67c6-449f-81a3-e5c53fd73e69","Type":"ContainerStarted","Data":"bbd7f97be3a60c487b3b969b0b7a0f43e9af9c079bd53eb2147c3f61336197b6"} Jul 21 22:28:25.383308 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:25.383251 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" podStartSLOduration=1.383231287 podStartE2EDuration="1.383231287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:28:25.383064111 +0000 UTC m=+2383.146298371" watchObservedRunningTime="2026-07-21 22:28:25.383231287 +0000 UTC m=+2383.146465550" Jul 21 22:28:25.838783 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:25.838750 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:26.844449 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:26.844417 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:27.376642 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:27.376607 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:27.377568 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:27.377550 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:28.825124 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.825100 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:28:28.877573 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.877491 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-data\") pod \"d4ecf4cf-1065-4529-9612-9b1e9d527204\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " Jul 21 22:28:28.877573 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.877530 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4ecf4cf-1065-4529-9612-9b1e9d527204-istiod-ca-cert\") pod \"d4ecf4cf-1065-4529-9612-9b1e9d527204\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " Jul 21 22:28:28.877573 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.877558 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-token\") pod \"d4ecf4cf-1065-4529-9612-9b1e9d527204\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " Jul 21 22:28:28.877861 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.877748 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-credential-socket\") pod \"d4ecf4cf-1065-4529-9612-9b1e9d527204\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " Jul 21 22:28:28.877861 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.877836 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-envoy\") pod \"d4ecf4cf-1065-4529-9612-9b1e9d527204\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " Jul 21 22:28:28.877975 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.877874 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-workload-socket\") pod \"d4ecf4cf-1065-4529-9612-9b1e9d527204\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " Jul 21 22:28:28.877975 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.877948 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-podinfo\") pod \"d4ecf4cf-1065-4529-9612-9b1e9d527204\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " Jul 21 22:28:28.878085 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.878001 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx2wt\" (UniqueName: \"kubernetes.io/projected/d4ecf4cf-1065-4529-9612-9b1e9d527204-kube-api-access-jx2wt\") pod \"d4ecf4cf-1065-4529-9612-9b1e9d527204\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " Jul 21 22:28:28.878085 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.878054 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-workload-certs\") pod \"d4ecf4cf-1065-4529-9612-9b1e9d527204\" (UID: \"d4ecf4cf-1065-4529-9612-9b1e9d527204\") " Jul 21 22:28:28.878397 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.877945 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ecf4cf-1065-4529-9612-9b1e9d527204-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d4ecf4cf-1065-4529-9612-9b1e9d527204" (UID: "d4ecf4cf-1065-4529-9612-9b1e9d527204"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:28:28.878575 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.878184 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d4ecf4cf-1065-4529-9612-9b1e9d527204" (UID: "d4ecf4cf-1065-4529-9612-9b1e9d527204"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:28:28.878718 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.878341 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d4ecf4cf-1065-4529-9612-9b1e9d527204" (UID: "d4ecf4cf-1065-4529-9612-9b1e9d527204"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:28:28.878811 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.878405 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-data" (OuterVolumeSpecName: "istio-data") pod "d4ecf4cf-1065-4529-9612-9b1e9d527204" (UID: "d4ecf4cf-1065-4529-9612-9b1e9d527204"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:28:28.879077 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.879037 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d4ecf4cf-1065-4529-9612-9b1e9d527204" (UID: "d4ecf4cf-1065-4529-9612-9b1e9d527204"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:28:28.880631 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.880591 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d4ecf4cf-1065-4529-9612-9b1e9d527204" (UID: "d4ecf4cf-1065-4529-9612-9b1e9d527204"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:28:28.880903 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.880871 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-token" (OuterVolumeSpecName: "istio-token") pod "d4ecf4cf-1065-4529-9612-9b1e9d527204" (UID: "d4ecf4cf-1065-4529-9612-9b1e9d527204"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:28:28.881068 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.881034 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ecf4cf-1065-4529-9612-9b1e9d527204-kube-api-access-jx2wt" (OuterVolumeSpecName: "kube-api-access-jx2wt") pod "d4ecf4cf-1065-4529-9612-9b1e9d527204" (UID: "d4ecf4cf-1065-4529-9612-9b1e9d527204"). InnerVolumeSpecName "kube-api-access-jx2wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:28:28.881315 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.881295 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d4ecf4cf-1065-4529-9612-9b1e9d527204" (UID: "d4ecf4cf-1065-4529-9612-9b1e9d527204"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 22:28:28.979741 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.979711 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-data\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:28:28.979741 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.979742 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4ecf4cf-1065-4529-9612-9b1e9d527204-istiod-ca-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:28:28.979960 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.979753 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-token\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:28:28.979960 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.979765 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-credential-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:28:28.979960 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.979778 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-envoy\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:28:28.979960 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.979791 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-workload-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:28:28.979960 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.979801 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4ecf4cf-1065-4529-9612-9b1e9d527204-istio-podinfo\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:28:28.979960 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.979813 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx2wt\" (UniqueName: \"kubernetes.io/projected/d4ecf4cf-1065-4529-9612-9b1e9d527204-kube-api-access-jx2wt\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:28:28.979960 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:28.979825 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4ecf4cf-1065-4529-9612-9b1e9d527204-workload-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:28:29.389048 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:29.389015 2580 generic.go:356] "Generic (PLEG): container finished" podID="d4ecf4cf-1065-4529-9612-9b1e9d527204" containerID="9d75edfc02d50994d3fabe73c8a4d3df523a9f28666bc36b417b4f10a5f8c1d4" exitCode=0 Jul 21 22:28:29.389213 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:29.389065 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" Jul 21 22:28:29.389213 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:29.389095 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" event={"ID":"d4ecf4cf-1065-4529-9612-9b1e9d527204","Type":"ContainerDied","Data":"9d75edfc02d50994d3fabe73c8a4d3df523a9f28666bc36b417b4f10a5f8c1d4"} Jul 21 22:28:29.389213 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:29.389144 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z" event={"ID":"d4ecf4cf-1065-4529-9612-9b1e9d527204","Type":"ContainerDied","Data":"fbbbd2613dcd6353539ac34f4040d5fa48b8b8840f4df4be74f560f81c690d23"} Jul 21 22:28:29.389213 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:29.389166 2580 scope.go:117] "RemoveContainer" containerID="9d75edfc02d50994d3fabe73c8a4d3df523a9f28666bc36b417b4f10a5f8c1d4" Jul 21 22:28:29.398749 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:29.398730 2580 scope.go:117] "RemoveContainer" containerID="09659ad7aedef76307c2f10f81626e86ce971e9b29b0775f4806afb06256aa5f" Jul 21 22:28:29.406914 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:29.406900 2580 scope.go:117] "RemoveContainer" containerID="9d75edfc02d50994d3fabe73c8a4d3df523a9f28666bc36b417b4f10a5f8c1d4" Jul 21 22:28:29.407140 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:28:29.407115 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d75edfc02d50994d3fabe73c8a4d3df523a9f28666bc36b417b4f10a5f8c1d4\": container with ID starting with 9d75edfc02d50994d3fabe73c8a4d3df523a9f28666bc36b417b4f10a5f8c1d4 not found: ID does not exist" containerID="9d75edfc02d50994d3fabe73c8a4d3df523a9f28666bc36b417b4f10a5f8c1d4" Jul 21 22:28:29.407185 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:29.407149 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d75edfc02d50994d3fabe73c8a4d3df523a9f28666bc36b417b4f10a5f8c1d4"} err="failed to get container status \"9d75edfc02d50994d3fabe73c8a4d3df523a9f28666bc36b417b4f10a5f8c1d4\": rpc error: code = NotFound desc = could not find container \"9d75edfc02d50994d3fabe73c8a4d3df523a9f28666bc36b417b4f10a5f8c1d4\": container with ID starting with 9d75edfc02d50994d3fabe73c8a4d3df523a9f28666bc36b417b4f10a5f8c1d4 not found: ID does not exist" Jul 21 22:28:29.407185 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:29.407164 2580 scope.go:117] "RemoveContainer" containerID="09659ad7aedef76307c2f10f81626e86ce971e9b29b0775f4806afb06256aa5f" Jul 21 22:28:29.407497 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:28:29.407477 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09659ad7aedef76307c2f10f81626e86ce971e9b29b0775f4806afb06256aa5f\": container with ID starting with 09659ad7aedef76307c2f10f81626e86ce971e9b29b0775f4806afb06256aa5f not found: ID does not exist" containerID="09659ad7aedef76307c2f10f81626e86ce971e9b29b0775f4806afb06256aa5f" Jul 21 22:28:29.407561 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:29.407502 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09659ad7aedef76307c2f10f81626e86ce971e9b29b0775f4806afb06256aa5f"} err="failed to get container status \"09659ad7aedef76307c2f10f81626e86ce971e9b29b0775f4806afb06256aa5f\": rpc error: code = NotFound desc = could not find container \"09659ad7aedef76307c2f10f81626e86ce971e9b29b0775f4806afb06256aa5f\": container with ID starting with 09659ad7aedef76307c2f10f81626e86ce971e9b29b0775f4806afb06256aa5f not found: ID does not exist" Jul 21 22:28:29.413499 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:29.413481 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z"] Jul 21 22:28:29.418241 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:29.418220 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-76e244-openshift-default-7b887489f7-fsp7z"] Jul 21 22:28:30.780497 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:30.780464 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4ecf4cf-1065-4529-9612-9b1e9d527204" path="/var/lib/kubelet/pods/d4ecf4cf-1065-4529-9612-9b1e9d527204/volumes" Jul 21 22:28:32.435327 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.435283 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99"] Jul 21 22:28:32.435887 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.435861 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="303027a2-5fed-4fc4-a554-29a28e88f2f6" containerName="istio-proxy" Jul 21 22:28:32.435887 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.435886 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="303027a2-5fed-4fc4-a554-29a28e88f2f6" containerName="istio-proxy" Jul 21 22:28:32.436081 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.435919 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ecf4cf-1065-4529-9612-9b1e9d527204" containerName="istio-proxy" Jul 21 22:28:32.436081 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.435925 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ecf4cf-1065-4529-9612-9b1e9d527204" containerName="istio-proxy" Jul 21 22:28:32.436081 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.435934 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ecf4cf-1065-4529-9612-9b1e9d527204" containerName="istio-proxy" Jul 21 22:28:32.436081 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.435940 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ecf4cf-1065-4529-9612-9b1e9d527204" containerName="istio-proxy" Jul 21 22:28:32.436081 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.436023 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4ecf4cf-1065-4529-9612-9b1e9d527204" containerName="istio-proxy" Jul 21 22:28:32.436081 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.436037 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4ecf4cf-1065-4529-9612-9b1e9d527204" containerName="istio-proxy" Jul 21 22:28:32.440532 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.440514 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.442971 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.442954 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-b77603-openshift-default-dockercfg-zbtfm\"" Jul 21 22:28:32.452241 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.452215 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99"] Jul 21 22:28:32.518582 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.518550 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvx9w\" (UniqueName: \"kubernetes.io/projected/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-kube-api-access-kvx9w\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.518741 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.518591 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-podinfo\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.518741 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.518732 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-token\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.518822 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.518777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istiod-ca-cert\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.518857 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.518820 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-credential-socket\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.518857 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.518841 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-data\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.518922 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.518898 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-workload-socket\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.518922 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.518913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-envoy\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.518988 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.518940 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-workload-certs\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.619666 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.619633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-workload-certs\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.619899 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.619698 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvx9w\" (UniqueName: \"kubernetes.io/projected/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-kube-api-access-kvx9w\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.619899 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.619737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-podinfo\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.619899 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.619830 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-token\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.620082 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.619905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istiod-ca-cert\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.620082 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.619939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-credential-socket\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.620082 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.619970 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-data\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.620082 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.620029 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-workload-socket\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.620082 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.620057 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-envoy\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.620421 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.620396 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-credential-socket\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.620521 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.620490 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-workload-certs\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.620921 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.620558 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-workload-socket\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.620921 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.620718 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-data\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.621191 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.621017 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istiod-ca-cert\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.622405 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.622365 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-envoy\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.622541 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.622522 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-podinfo\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.629253 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.629231 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-token\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.629907 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.629886 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvx9w\" (UniqueName: \"kubernetes.io/projected/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-kube-api-access-kvx9w\") pod \"e2e-shared-b-b77603-openshift-default-8758fd895-sqx99\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.752166 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.752125 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:32.943196 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.943166 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99"] Jul 21 22:28:32.945219 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:28:32.945194 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5531b6a1_3d90_4ed9_be46_ed8b43b7ea85.slice/crio-20e292f55adc20ffcad5b5928c6fb4a924cd4cee1ac31dcb9738cb08e6ad8694 WatchSource:0}: Error finding container 20e292f55adc20ffcad5b5928c6fb4a924cd4cee1ac31dcb9738cb08e6ad8694: Status 404 returned error can't find the container with id 20e292f55adc20ffcad5b5928c6fb4a924cd4cee1ac31dcb9738cb08e6ad8694 Jul 21 22:28:32.947344 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.947310 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:28:32.947447 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.947402 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:28:32.947447 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:32.947434 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:28:33.414784 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:33.414686 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" event={"ID":"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85","Type":"ContainerStarted","Data":"797c6ccbac57cb3b4fbb13ac5dccd6dd792647de67b902b49bc81aa53008b798"} Jul 21 22:28:33.414784 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:33.414755 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" event={"ID":"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85","Type":"ContainerStarted","Data":"20e292f55adc20ffcad5b5928c6fb4a924cd4cee1ac31dcb9738cb08e6ad8694"} Jul 21 22:28:33.439274 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:33.439220 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" podStartSLOduration=1.439205124 podStartE2EDuration="1.439205124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:28:33.436862178 +0000 UTC m=+2391.200096451" watchObservedRunningTime="2026-07-21 22:28:33.439205124 +0000 UTC m=+2391.202439386" Jul 21 22:28:33.752903 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:33.752875 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:34.757954 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:34.757923 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:35.425683 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:35.425643 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:35.426667 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:35.426625 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:40.453502 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:40.453467 2580 generic.go:356] "Generic (PLEG): container finished" podID="aeb92da7-67c6-449f-81a3-e5c53fd73e69" containerID="8220a49ab3c748d49de60c14fcbefc48a8a7a80d4e9bc808a86f3d031710f07a" exitCode=0 Jul 21 22:28:40.453915 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:40.453538 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" event={"ID":"aeb92da7-67c6-449f-81a3-e5c53fd73e69","Type":"ContainerDied","Data":"8220a49ab3c748d49de60c14fcbefc48a8a7a80d4e9bc808a86f3d031710f07a"} Jul 21 22:28:40.453972 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:40.453955 2580 scope.go:117] "RemoveContainer" containerID="8220a49ab3c748d49de60c14fcbefc48a8a7a80d4e9bc808a86f3d031710f07a" Jul 21 22:28:40.454983 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:40.454953 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:28:40.455065 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:40.455043 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:28:40.455106 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:40.455095 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:28:41.460430 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:41.460395 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" event={"ID":"aeb92da7-67c6-449f-81a3-e5c53fd73e69","Type":"ContainerStarted","Data":"5a29eb3de84cb432ed230eacd05b1e2f8939b8648f69486a17d08da93c020101"} Jul 21 22:28:41.838590 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:41.838552 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:41.839930 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:41.839899 2580 patch_prober.go:28] interesting pod/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 21 22:28:41.840078 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:41.839960 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" podUID="aeb92da7-67c6-449f-81a3-e5c53fd73e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 21 22:28:42.377000 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:42.376954 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:42.838672 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:42.838642 2580 patch_prober.go:28] interesting pod/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 21 22:28:42.839122 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:42.838689 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" podUID="aeb92da7-67c6-449f-81a3-e5c53fd73e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 21 22:28:43.838753 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:43.838714 2580 patch_prober.go:28] interesting pod/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 21 22:28:43.839154 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:43.838779 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" podUID="aeb92da7-67c6-449f-81a3-e5c53fd73e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 21 22:28:44.839142 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:44.839102 2580 patch_prober.go:28] interesting pod/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 21 22:28:44.839579 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:44.839158 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" podUID="aeb92da7-67c6-449f-81a3-e5c53fd73e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 21 22:28:45.843125 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:45.843092 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:45.844207 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:45.844188 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:28:50.427320 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:50.427285 2580 patch_prober.go:28] interesting pod/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 21 22:28:50.427790 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:50.427337 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" podUID="5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 21 22:28:50.519279 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:50.519243 2580 generic.go:356] "Generic (PLEG): container finished" podID="5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" containerID="797c6ccbac57cb3b4fbb13ac5dccd6dd792647de67b902b49bc81aa53008b798" exitCode=0 Jul 21 22:28:50.519479 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:50.519288 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" event={"ID":"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85","Type":"ContainerDied","Data":"797c6ccbac57cb3b4fbb13ac5dccd6dd792647de67b902b49bc81aa53008b798"} Jul 21 22:28:50.519736 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:50.519721 2580 scope.go:117] "RemoveContainer" containerID="797c6ccbac57cb3b4fbb13ac5dccd6dd792647de67b902b49bc81aa53008b798" Jul 21 22:28:50.520853 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:50.520811 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:28:50.520914 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:50.520902 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:28:50.520950 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:50.520931 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 22:28:51.526361 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:51.526328 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" event={"ID":"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85","Type":"ContainerStarted","Data":"ea61c3520feab0e4291f0e47af8c156970d074cc31901cf642e1d26160c038c8"} Jul 21 22:28:51.753044 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:51.753009 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:51.753214 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:51.753167 2580 patch_prober.go:28] interesting pod/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 21 22:28:51.753259 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:51.753215 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" podUID="5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 21 22:28:52.752949 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:52.752918 2580 patch_prober.go:28] interesting pod/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 21 22:28:52.753341 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:52.752968 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" podUID="5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 21 22:28:53.753082 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:53.753043 2580 patch_prober.go:28] interesting pod/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 21 22:28:53.753534 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:53.753108 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" podUID="5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 21 22:28:54.753467 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:54.753432 2580 patch_prober.go:28] interesting pod/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 21 22:28:54.753977 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:54.753490 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" podUID="5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 21 22:28:55.756552 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:55.756519 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:55.757003 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:55.756765 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:28:55.757607 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:28:55.757590 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:31:33.986475 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:33.986440 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b"] Jul 21 22:31:33.990197 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:33.990180 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:33.993939 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:33.993911 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-131f31\"/\"kube-root-ca.crt\"" Jul 21 22:31:33.993939 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:33.993928 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-131f31\"/\"openshift-service-ca.crt\"" Jul 21 22:31:33.994086 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:33.993964 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-131f31\"/\"sub-test-model-131f31-kserve-self-signed-certs\"" Jul 21 22:31:33.994086 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:33.993911 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-131f31\"/\"default-dockercfg-d8nnn\"" Jul 21 22:31:34.002130 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.002106 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b"] Jul 21 22:31:34.100538 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.100509 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-home\") pod \"sub-test-model-131f31-kserve-79dd444d77-zbz6b\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.100709 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.100555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-459ml\" (UniqueName: \"kubernetes.io/projected/4d544896-0303-4afa-b880-266f1f78ac8d-kube-api-access-459ml\") pod \"sub-test-model-131f31-kserve-79dd444d77-zbz6b\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.100709 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.100649 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-model-cache\") pod \"sub-test-model-131f31-kserve-79dd444d77-zbz6b\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.100709 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.100685 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-dshm\") pod \"sub-test-model-131f31-kserve-79dd444d77-zbz6b\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.100814 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.100787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-tmp-dir\") pod \"sub-test-model-131f31-kserve-79dd444d77-zbz6b\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.100853 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.100818 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d544896-0303-4afa-b880-266f1f78ac8d-tls-certs\") pod \"sub-test-model-131f31-kserve-79dd444d77-zbz6b\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.201919 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.201890 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-tmp-dir\") pod \"sub-test-model-131f31-kserve-79dd444d77-zbz6b\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.201919 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.201923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d544896-0303-4afa-b880-266f1f78ac8d-tls-certs\") pod \"sub-test-model-131f31-kserve-79dd444d77-zbz6b\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.202133 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.201941 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-home\") pod \"sub-test-model-131f31-kserve-79dd444d77-zbz6b\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.202133 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.201969 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-459ml\" (UniqueName: \"kubernetes.io/projected/4d544896-0303-4afa-b880-266f1f78ac8d-kube-api-access-459ml\") pod \"sub-test-model-131f31-kserve-79dd444d77-zbz6b\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.202133 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.202002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-model-cache\") pod \"sub-test-model-131f31-kserve-79dd444d77-zbz6b\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.202133 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.202025 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-dshm\") pod \"sub-test-model-131f31-kserve-79dd444d77-zbz6b\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.202353 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.202286 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-tmp-dir\") pod \"sub-test-model-131f31-kserve-79dd444d77-zbz6b\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.202447 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.202364 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-home\") pod \"sub-test-model-131f31-kserve-79dd444d77-zbz6b\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.202495 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.202447 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-model-cache\") pod \"sub-test-model-131f31-kserve-79dd444d77-zbz6b\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.204292 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.204265 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-dshm\") pod \"sub-test-model-131f31-kserve-79dd444d77-zbz6b\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.204554 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.204538 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d544896-0303-4afa-b880-266f1f78ac8d-tls-certs\") pod \"sub-test-model-131f31-kserve-79dd444d77-zbz6b\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.210812 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.210790 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-459ml\" (UniqueName: \"kubernetes.io/projected/4d544896-0303-4afa-b880-266f1f78ac8d-kube-api-access-459ml\") pod \"sub-test-model-131f31-kserve-79dd444d77-zbz6b\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.300703 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.300606 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:34.440315 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:34.440291 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b"] Jul 21 22:31:34.442030 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:31:34.442004 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d544896_0303_4afa_b880_266f1f78ac8d.slice/crio-8dd9f5b3a6ae56683c5e6984058c3117ed13181d641d1baf551e062e006f92ec WatchSource:0}: Error finding container 8dd9f5b3a6ae56683c5e6984058c3117ed13181d641d1baf551e062e006f92ec: Status 404 returned error can't find the container with id 8dd9f5b3a6ae56683c5e6984058c3117ed13181d641d1baf551e062e006f92ec Jul 21 22:31:35.404752 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:35.404713 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" event={"ID":"4d544896-0303-4afa-b880-266f1f78ac8d","Type":"ContainerStarted","Data":"1d5a4b79c754050b92f09f9adbd0f06451419b97070ddb9ec10d5ea00a8adc8a"} Jul 21 22:31:35.405139 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:35.404765 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" event={"ID":"4d544896-0303-4afa-b880-266f1f78ac8d","Type":"ContainerStarted","Data":"8dd9f5b3a6ae56683c5e6984058c3117ed13181d641d1baf551e062e006f92ec"} Jul 21 22:31:35.424016 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:35.423971 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" podStartSLOduration=2.423956909 podStartE2EDuration="2.423956909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:31:35.421661011 +0000 UTC m=+2573.184895275" watchObservedRunningTime="2026-07-21 22:31:35.423956909 +0000 UTC m=+2573.187191173" Jul 21 22:31:44.301676 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:44.301640 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:44.302059 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:44.301700 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:44.314182 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:44.314159 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:31:44.463615 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:31:44.463580 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:32:47.600593 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:32:47.600547 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b"] Jul 21 22:33:02.621757 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.621690 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" podUID="4d544896-0303-4afa-b880-266f1f78ac8d" containerName="main" containerID="cri-o://1d5a4b79c754050b92f09f9adbd0f06451419b97070ddb9ec10d5ea00a8adc8a" gracePeriod=45 Jul 21 22:33:02.877578 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.877499 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d544896-0303-4afa-b880-266f1f78ac8d" containerID="1d5a4b79c754050b92f09f9adbd0f06451419b97070ddb9ec10d5ea00a8adc8a" exitCode=0 Jul 21 22:33:02.877700 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.877579 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" event={"ID":"4d544896-0303-4afa-b880-266f1f78ac8d","Type":"ContainerDied","Data":"1d5a4b79c754050b92f09f9adbd0f06451419b97070ddb9ec10d5ea00a8adc8a"} Jul 21 22:33:02.877700 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.877622 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" event={"ID":"4d544896-0303-4afa-b880-266f1f78ac8d","Type":"ContainerDied","Data":"8dd9f5b3a6ae56683c5e6984058c3117ed13181d641d1baf551e062e006f92ec"} Jul 21 22:33:02.877700 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.877634 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dd9f5b3a6ae56683c5e6984058c3117ed13181d641d1baf551e062e006f92ec" Jul 21 22:33:02.879768 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.879754 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:33:02.892673 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.892653 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-tmp-dir\") pod \"4d544896-0303-4afa-b880-266f1f78ac8d\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " Jul 21 22:33:02.892765 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.892715 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-459ml\" (UniqueName: \"kubernetes.io/projected/4d544896-0303-4afa-b880-266f1f78ac8d-kube-api-access-459ml\") pod \"4d544896-0303-4afa-b880-266f1f78ac8d\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " Jul 21 22:33:02.892765 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.892731 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-dshm\") pod \"4d544896-0303-4afa-b880-266f1f78ac8d\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " Jul 21 22:33:02.892873 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.892780 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d544896-0303-4afa-b880-266f1f78ac8d-tls-certs\") pod \"4d544896-0303-4afa-b880-266f1f78ac8d\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " Jul 21 22:33:02.892873 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.892802 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-model-cache\") pod \"4d544896-0303-4afa-b880-266f1f78ac8d\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " Jul 21 22:33:02.892873 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.892837 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-home\") pod \"4d544896-0303-4afa-b880-266f1f78ac8d\" (UID: \"4d544896-0303-4afa-b880-266f1f78ac8d\") " Jul 21 22:33:02.893030 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.892898 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d544896-0303-4afa-b880-266f1f78ac8d" (UID: "4d544896-0303-4afa-b880-266f1f78ac8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:33:02.893173 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.893137 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d544896-0303-4afa-b880-266f1f78ac8d" (UID: "4d544896-0303-4afa-b880-266f1f78ac8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:33:02.893173 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.893149 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-home" (OuterVolumeSpecName: "home") pod "4d544896-0303-4afa-b880-266f1f78ac8d" (UID: "4d544896-0303-4afa-b880-266f1f78ac8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:33:02.893328 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.893276 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-model-cache\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:33:02.893328 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.893292 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-home\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:33:02.893328 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.893302 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-tmp-dir\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:33:02.895133 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.895109 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d544896-0303-4afa-b880-266f1f78ac8d-kube-api-access-459ml" (OuterVolumeSpecName: "kube-api-access-459ml") pod "4d544896-0303-4afa-b880-266f1f78ac8d" (UID: "4d544896-0303-4afa-b880-266f1f78ac8d"). InnerVolumeSpecName "kube-api-access-459ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:33:02.895523 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.895501 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d544896-0303-4afa-b880-266f1f78ac8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d544896-0303-4afa-b880-266f1f78ac8d" (UID: "4d544896-0303-4afa-b880-266f1f78ac8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:33:02.895618 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.895501 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-dshm" (OuterVolumeSpecName: "dshm") pod "4d544896-0303-4afa-b880-266f1f78ac8d" (UID: "4d544896-0303-4afa-b880-266f1f78ac8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:33:02.994594 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.994561 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d544896-0303-4afa-b880-266f1f78ac8d-tls-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:33:02.994594 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.994589 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-459ml\" (UniqueName: \"kubernetes.io/projected/4d544896-0303-4afa-b880-266f1f78ac8d-kube-api-access-459ml\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:33:02.994594 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:02.994600 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d544896-0303-4afa-b880-266f1f78ac8d-dshm\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:33:03.882762 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:03.882731 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b" Jul 21 22:33:03.904824 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:03.904796 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b"] Jul 21 22:33:03.910905 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:03.910882 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-131f31/sub-test-model-131f31-kserve-79dd444d77-zbz6b"] Jul 21 22:33:04.780515 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:04.780471 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d544896-0303-4afa-b880-266f1f78ac8d" path="/var/lib/kubelet/pods/4d544896-0303-4afa-b880-266f1f78ac8d/volumes" Jul 21 22:33:19.977165 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:19.977129 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm"] Jul 21 22:33:19.977894 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:19.977873 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d544896-0303-4afa-b880-266f1f78ac8d" containerName="main" Jul 21 22:33:19.977946 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:19.977899 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d544896-0303-4afa-b880-266f1f78ac8d" containerName="main" Jul 21 22:33:19.978036 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:19.978024 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d544896-0303-4afa-b880-266f1f78ac8d" containerName="main" Jul 21 22:33:19.981617 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:19.981595 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:19.985550 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:19.985528 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b77603\"/\"openshift-service-ca.crt\"" Jul 21 22:33:19.985686 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:19.985548 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b77603\"/\"default-dockercfg-p4lx2\"" Jul 21 22:33:19.985686 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:19.985594 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b77603\"/\"kube-root-ca.crt\"" Jul 21 22:33:19.985686 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:19.985593 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b77603\"/\"rate-test-model-b77603-kserve-self-signed-certs\"" Jul 21 22:33:19.990702 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:19.990654 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm"] Jul 21 22:33:20.066516 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.066476 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4n66\" (UniqueName: \"kubernetes.io/projected/ff10763a-6e60-4318-8c9f-95eb4565f372-kube-api-access-c4n66\") pod \"rate-test-model-b77603-kserve-68c48bc879-l86bm\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.066712 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.066590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-dshm\") pod \"rate-test-model-b77603-kserve-68c48bc879-l86bm\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.066712 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.066688 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff10763a-6e60-4318-8c9f-95eb4565f372-tls-certs\") pod \"rate-test-model-b77603-kserve-68c48bc879-l86bm\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.066847 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.066761 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-model-cache\") pod \"rate-test-model-b77603-kserve-68c48bc879-l86bm\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.066847 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.066787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-tmp-dir\") pod \"rate-test-model-b77603-kserve-68c48bc879-l86bm\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.066847 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.066805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-home\") pod \"rate-test-model-b77603-kserve-68c48bc879-l86bm\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.167644 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.167614 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-dshm\") pod \"rate-test-model-b77603-kserve-68c48bc879-l86bm\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.167824 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.167682 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff10763a-6e60-4318-8c9f-95eb4565f372-tls-certs\") pod \"rate-test-model-b77603-kserve-68c48bc879-l86bm\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.167824 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.167723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-model-cache\") pod \"rate-test-model-b77603-kserve-68c48bc879-l86bm\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.167824 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.167738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-tmp-dir\") pod \"rate-test-model-b77603-kserve-68c48bc879-l86bm\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.167824 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.167762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-home\") pod \"rate-test-model-b77603-kserve-68c48bc879-l86bm\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.168046 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.167835 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4n66\" (UniqueName: \"kubernetes.io/projected/ff10763a-6e60-4318-8c9f-95eb4565f372-kube-api-access-c4n66\") pod \"rate-test-model-b77603-kserve-68c48bc879-l86bm\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.168129 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.168105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-model-cache\") pod \"rate-test-model-b77603-kserve-68c48bc879-l86bm\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.168192 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.168144 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-tmp-dir\") pod \"rate-test-model-b77603-kserve-68c48bc879-l86bm\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.168247 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.168186 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-home\") pod \"rate-test-model-b77603-kserve-68c48bc879-l86bm\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.169990 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.169970 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-dshm\") pod \"rate-test-model-b77603-kserve-68c48bc879-l86bm\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.170361 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.170339 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff10763a-6e60-4318-8c9f-95eb4565f372-tls-certs\") pod \"rate-test-model-b77603-kserve-68c48bc879-l86bm\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.178517 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.178472 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4n66\" (UniqueName: \"kubernetes.io/projected/ff10763a-6e60-4318-8c9f-95eb4565f372-kube-api-access-c4n66\") pod \"rate-test-model-b77603-kserve-68c48bc879-l86bm\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.293473 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.293430 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:20.439588 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.439549 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm"] Jul 21 22:33:20.441668 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:33:20.441640 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff10763a_6e60_4318_8c9f_95eb4565f372.slice/crio-ffb6aba489c771dccea006916e468f1a46ffa915475964790d04a264aee301b2 WatchSource:0}: Error finding container ffb6aba489c771dccea006916e468f1a46ffa915475964790d04a264aee301b2: Status 404 returned error can't find the container with id ffb6aba489c771dccea006916e468f1a46ffa915475964790d04a264aee301b2 Jul 21 22:33:20.977094 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.977055 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" event={"ID":"ff10763a-6e60-4318-8c9f-95eb4565f372","Type":"ContainerStarted","Data":"c771221102e9c50e013992a0c4a6ec24f13f6f070f88e009485442464075d4a7"} Jul 21 22:33:20.977276 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.977106 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" event={"ID":"ff10763a-6e60-4318-8c9f-95eb4565f372","Type":"ContainerStarted","Data":"ffb6aba489c771dccea006916e468f1a46ffa915475964790d04a264aee301b2"} Jul 21 22:33:20.998445 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:20.998399 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" podStartSLOduration=1.9983673309999999 podStartE2EDuration="1.998367331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:33:20.996316055 +0000 UTC m=+2678.759550340" watchObservedRunningTime="2026-07-21 22:33:20.998367331 +0000 UTC m=+2678.761601591" Jul 21 22:33:30.294276 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:30.294242 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:30.294763 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:30.294329 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:30.306359 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:30.306334 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:33:31.045804 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:33:31.045774 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:34:03.503822 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:03.503787 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm"] Jul 21 22:34:18.524768 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.524699 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" podUID="ff10763a-6e60-4318-8c9f-95eb4565f372" containerName="main" containerID="cri-o://c771221102e9c50e013992a0c4a6ec24f13f6f070f88e009485442464075d4a7" gracePeriod=45 Jul 21 22:34:18.774989 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.774926 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:34:18.862017 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.861987 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-tmp-dir\") pod \"ff10763a-6e60-4318-8c9f-95eb4565f372\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " Jul 21 22:34:18.862157 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.862040 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4n66\" (UniqueName: \"kubernetes.io/projected/ff10763a-6e60-4318-8c9f-95eb4565f372-kube-api-access-c4n66\") pod \"ff10763a-6e60-4318-8c9f-95eb4565f372\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " Jul 21 22:34:18.862157 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.862086 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff10763a-6e60-4318-8c9f-95eb4565f372-tls-certs\") pod \"ff10763a-6e60-4318-8c9f-95eb4565f372\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " Jul 21 22:34:18.862157 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.862128 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-dshm\") pod \"ff10763a-6e60-4318-8c9f-95eb4565f372\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " Jul 21 22:34:18.862157 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.862157 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-model-cache\") pod \"ff10763a-6e60-4318-8c9f-95eb4565f372\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " Jul 21 22:34:18.862355 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.862186 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-home\") pod \"ff10763a-6e60-4318-8c9f-95eb4565f372\" (UID: \"ff10763a-6e60-4318-8c9f-95eb4565f372\") " Jul 21 22:34:18.862355 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.862242 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff10763a-6e60-4318-8c9f-95eb4565f372" (UID: "ff10763a-6e60-4318-8c9f-95eb4565f372"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:34:18.862498 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.862453 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff10763a-6e60-4318-8c9f-95eb4565f372" (UID: "ff10763a-6e60-4318-8c9f-95eb4565f372"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:34:18.862556 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.862525 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-home" (OuterVolumeSpecName: "home") pod "ff10763a-6e60-4318-8c9f-95eb4565f372" (UID: "ff10763a-6e60-4318-8c9f-95eb4565f372"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:34:18.862632 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.862616 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-tmp-dir\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:34:18.862679 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.862637 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-model-cache\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:34:18.862679 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.862649 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-home\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:34:18.864345 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.864320 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-dshm" (OuterVolumeSpecName: "dshm") pod "ff10763a-6e60-4318-8c9f-95eb4565f372" (UID: "ff10763a-6e60-4318-8c9f-95eb4565f372"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:34:18.864476 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.864405 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff10763a-6e60-4318-8c9f-95eb4565f372-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff10763a-6e60-4318-8c9f-95eb4565f372" (UID: "ff10763a-6e60-4318-8c9f-95eb4565f372"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:34:18.864476 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.864421 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff10763a-6e60-4318-8c9f-95eb4565f372-kube-api-access-c4n66" (OuterVolumeSpecName: "kube-api-access-c4n66") pod "ff10763a-6e60-4318-8c9f-95eb4565f372" (UID: "ff10763a-6e60-4318-8c9f-95eb4565f372"). InnerVolumeSpecName "kube-api-access-c4n66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:34:18.963659 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.963624 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff10763a-6e60-4318-8c9f-95eb4565f372-tls-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:34:18.963659 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.963655 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff10763a-6e60-4318-8c9f-95eb4565f372-dshm\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:34:18.963659 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:18.963666 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4n66\" (UniqueName: \"kubernetes.io/projected/ff10763a-6e60-4318-8c9f-95eb4565f372-kube-api-access-c4n66\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:34:19.291792 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:19.291759 2580 generic.go:356] "Generic (PLEG): container finished" podID="ff10763a-6e60-4318-8c9f-95eb4565f372" containerID="c771221102e9c50e013992a0c4a6ec24f13f6f070f88e009485442464075d4a7" exitCode=0 Jul 21 22:34:19.291982 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:19.291802 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" Jul 21 22:34:19.291982 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:19.291851 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" event={"ID":"ff10763a-6e60-4318-8c9f-95eb4565f372","Type":"ContainerDied","Data":"c771221102e9c50e013992a0c4a6ec24f13f6f070f88e009485442464075d4a7"} Jul 21 22:34:19.291982 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:19.291901 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm" event={"ID":"ff10763a-6e60-4318-8c9f-95eb4565f372","Type":"ContainerDied","Data":"ffb6aba489c771dccea006916e468f1a46ffa915475964790d04a264aee301b2"} Jul 21 22:34:19.291982 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:19.291923 2580 scope.go:117] "RemoveContainer" containerID="c771221102e9c50e013992a0c4a6ec24f13f6f070f88e009485442464075d4a7" Jul 21 22:34:19.301794 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:19.301780 2580 scope.go:117] "RemoveContainer" containerID="c771221102e9c50e013992a0c4a6ec24f13f6f070f88e009485442464075d4a7" Jul 21 22:34:19.302022 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:34:19.302005 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c771221102e9c50e013992a0c4a6ec24f13f6f070f88e009485442464075d4a7\": container with ID starting with c771221102e9c50e013992a0c4a6ec24f13f6f070f88e009485442464075d4a7 not found: ID does not exist" containerID="c771221102e9c50e013992a0c4a6ec24f13f6f070f88e009485442464075d4a7" Jul 21 22:34:19.302087 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:19.302033 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c771221102e9c50e013992a0c4a6ec24f13f6f070f88e009485442464075d4a7"} err="failed to get container status \"c771221102e9c50e013992a0c4a6ec24f13f6f070f88e009485442464075d4a7\": rpc error: code = NotFound desc = could not find container \"c771221102e9c50e013992a0c4a6ec24f13f6f070f88e009485442464075d4a7\": container with ID starting with c771221102e9c50e013992a0c4a6ec24f13f6f070f88e009485442464075d4a7 not found: ID does not exist" Jul 21 22:34:19.314095 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:19.314075 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm"] Jul 21 22:34:19.317559 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:19.317541 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b77603/rate-test-model-b77603-kserve-68c48bc879-l86bm"] Jul 21 22:34:20.782406 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:20.782346 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff10763a-6e60-4318-8c9f-95eb4565f372" path="/var/lib/kubelet/pods/ff10763a-6e60-4318-8c9f-95eb4565f372/volumes" Jul 21 22:34:44.678070 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:44.678037 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6"] Jul 21 22:34:44.678608 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:44.678273 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" podUID="aeb92da7-67c6-449f-81a3-e5c53fd73e69" containerName="istio-proxy" containerID="cri-o://5a29eb3de84cb432ed230eacd05b1e2f8939b8648f69486a17d08da93c020101" gracePeriod=30 Jul 21 22:34:45.845899 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:45.845859 2580 patch_prober.go:28] interesting pod/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 22:34:45.846294 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:45.845919 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" podUID="aeb92da7-67c6-449f-81a3-e5c53fd73e69" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 22:34:49.934402 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:49.934360 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:34:50.066025 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.065996 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-data\") pod \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " Jul 21 22:34:50.066194 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.066035 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-workload-certs\") pod \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " Jul 21 22:34:50.066194 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.066069 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-podinfo\") pod \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " Jul 21 22:34:50.066194 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.066114 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-workload-socket\") pod \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " Jul 21 22:34:50.066194 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.066132 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-envoy\") pod \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " Jul 21 22:34:50.066419 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.066224 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-credential-socket\") pod \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " Jul 21 22:34:50.066419 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.066313 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6vvg\" (UniqueName: \"kubernetes.io/projected/aeb92da7-67c6-449f-81a3-e5c53fd73e69-kube-api-access-p6vvg\") pod \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " Jul 21 22:34:50.066419 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.066323 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aeb92da7-67c6-449f-81a3-e5c53fd73e69" (UID: "aeb92da7-67c6-449f-81a3-e5c53fd73e69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:34:50.066419 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.066344 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-token\") pod \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " Jul 21 22:34:50.066419 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.066368 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istiod-ca-cert\") pod \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\" (UID: \"aeb92da7-67c6-449f-81a3-e5c53fd73e69\") " Jul 21 22:34:50.066685 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.066482 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aeb92da7-67c6-449f-81a3-e5c53fd73e69" (UID: "aeb92da7-67c6-449f-81a3-e5c53fd73e69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:34:50.066685 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.066536 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aeb92da7-67c6-449f-81a3-e5c53fd73e69" (UID: "aeb92da7-67c6-449f-81a3-e5c53fd73e69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:34:50.066789 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.066763 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-workload-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:34:50.066789 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.066783 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-credential-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:34:50.066889 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.066789 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aeb92da7-67c6-449f-81a3-e5c53fd73e69" (UID: "aeb92da7-67c6-449f-81a3-e5c53fd73e69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:34:50.066889 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.066800 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-workload-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:34:50.067236 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.067210 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-data" (OuterVolumeSpecName: "istio-data") pod "aeb92da7-67c6-449f-81a3-e5c53fd73e69" (UID: "aeb92da7-67c6-449f-81a3-e5c53fd73e69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:34:50.068697 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.068670 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb92da7-67c6-449f-81a3-e5c53fd73e69-kube-api-access-p6vvg" (OuterVolumeSpecName: "kube-api-access-p6vvg") pod "aeb92da7-67c6-449f-81a3-e5c53fd73e69" (UID: "aeb92da7-67c6-449f-81a3-e5c53fd73e69"). InnerVolumeSpecName "kube-api-access-p6vvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:34:50.068793 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.068675 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-token" (OuterVolumeSpecName: "istio-token") pod "aeb92da7-67c6-449f-81a3-e5c53fd73e69" (UID: "aeb92da7-67c6-449f-81a3-e5c53fd73e69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:34:50.068917 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.068895 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aeb92da7-67c6-449f-81a3-e5c53fd73e69" (UID: "aeb92da7-67c6-449f-81a3-e5c53fd73e69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 22:34:50.068917 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.068901 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aeb92da7-67c6-449f-81a3-e5c53fd73e69" (UID: "aeb92da7-67c6-449f-81a3-e5c53fd73e69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:34:50.168347 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.168317 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-data\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:34:50.168347 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.168344 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-podinfo\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:34:50.168347 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.168354 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-envoy\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:34:50.168586 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.168363 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6vvg\" (UniqueName: \"kubernetes.io/projected/aeb92da7-67c6-449f-81a3-e5c53fd73e69-kube-api-access-p6vvg\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:34:50.168586 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.168390 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istio-token\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:34:50.168586 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.168402 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeb92da7-67c6-449f-81a3-e5c53fd73e69-istiod-ca-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:34:50.467139 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.467042 2580 generic.go:356] "Generic (PLEG): container finished" podID="aeb92da7-67c6-449f-81a3-e5c53fd73e69" containerID="5a29eb3de84cb432ed230eacd05b1e2f8939b8648f69486a17d08da93c020101" exitCode=0 Jul 21 22:34:50.467139 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.467096 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" Jul 21 22:34:50.467394 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.467133 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" event={"ID":"aeb92da7-67c6-449f-81a3-e5c53fd73e69","Type":"ContainerDied","Data":"5a29eb3de84cb432ed230eacd05b1e2f8939b8648f69486a17d08da93c020101"} Jul 21 22:34:50.467394 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.467203 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6" event={"ID":"aeb92da7-67c6-449f-81a3-e5c53fd73e69","Type":"ContainerDied","Data":"bbd7f97be3a60c487b3b969b0b7a0f43e9af9c079bd53eb2147c3f61336197b6"} Jul 21 22:34:50.467394 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.467235 2580 scope.go:117] "RemoveContainer" containerID="5a29eb3de84cb432ed230eacd05b1e2f8939b8648f69486a17d08da93c020101" Jul 21 22:34:50.477068 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.477050 2580 scope.go:117] "RemoveContainer" containerID="8220a49ab3c748d49de60c14fcbefc48a8a7a80d4e9bc808a86f3d031710f07a" Jul 21 22:34:50.485485 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.485448 2580 scope.go:117] "RemoveContainer" containerID="5a29eb3de84cb432ed230eacd05b1e2f8939b8648f69486a17d08da93c020101" Jul 21 22:34:50.485735 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:34:50.485709 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a29eb3de84cb432ed230eacd05b1e2f8939b8648f69486a17d08da93c020101\": container with ID starting with 5a29eb3de84cb432ed230eacd05b1e2f8939b8648f69486a17d08da93c020101 not found: ID does not exist" containerID="5a29eb3de84cb432ed230eacd05b1e2f8939b8648f69486a17d08da93c020101" Jul 21 22:34:50.485818 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.485748 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a29eb3de84cb432ed230eacd05b1e2f8939b8648f69486a17d08da93c020101"} err="failed to get container status \"5a29eb3de84cb432ed230eacd05b1e2f8939b8648f69486a17d08da93c020101\": rpc error: code = NotFound desc = could not find container \"5a29eb3de84cb432ed230eacd05b1e2f8939b8648f69486a17d08da93c020101\": container with ID starting with 5a29eb3de84cb432ed230eacd05b1e2f8939b8648f69486a17d08da93c020101 not found: ID does not exist" Jul 21 22:34:50.485818 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.485771 2580 scope.go:117] "RemoveContainer" containerID="8220a49ab3c748d49de60c14fcbefc48a8a7a80d4e9bc808a86f3d031710f07a" Jul 21 22:34:50.486073 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:34:50.486048 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8220a49ab3c748d49de60c14fcbefc48a8a7a80d4e9bc808a86f3d031710f07a\": container with ID starting with 8220a49ab3c748d49de60c14fcbefc48a8a7a80d4e9bc808a86f3d031710f07a not found: ID does not exist" containerID="8220a49ab3c748d49de60c14fcbefc48a8a7a80d4e9bc808a86f3d031710f07a" Jul 21 22:34:50.486144 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.486082 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8220a49ab3c748d49de60c14fcbefc48a8a7a80d4e9bc808a86f3d031710f07a"} err="failed to get container status \"8220a49ab3c748d49de60c14fcbefc48a8a7a80d4e9bc808a86f3d031710f07a\": rpc error: code = NotFound desc = could not find container \"8220a49ab3c748d49de60c14fcbefc48a8a7a80d4e9bc808a86f3d031710f07a\": container with ID starting with 8220a49ab3c748d49de60c14fcbefc48a8a7a80d4e9bc808a86f3d031710f07a not found: ID does not exist" Jul 21 22:34:50.490446 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.490427 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6"] Jul 21 22:34:50.494832 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.494813 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-131f31-openshift-default-5d599fd47-jk7n6"] Jul 21 22:34:50.780453 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:34:50.780425 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeb92da7-67c6-449f-81a3-e5c53fd73e69" path="/var/lib/kubelet/pods/aeb92da7-67c6-449f-81a3-e5c53fd73e69/volumes" Jul 21 22:35:07.057422 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:07.057362 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99"] Jul 21 22:35:07.057924 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:07.057647 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" podUID="5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" containerName="istio-proxy" containerID="cri-o://ea61c3520feab0e4291f0e47af8c156970d074cc31901cf642e1d26160c038c8" gracePeriod=30 Jul 21 22:35:10.759083 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:10.759046 2580 patch_prober.go:28] interesting pod/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 22:35:10.759578 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:10.759115 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" podUID="5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 22:35:12.322288 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.322236 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:35:12.391012 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.390988 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-token\") pod \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " Jul 21 22:35:12.391139 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.391022 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvx9w\" (UniqueName: \"kubernetes.io/projected/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-kube-api-access-kvx9w\") pod \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " Jul 21 22:35:12.391139 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.391046 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-credential-socket\") pod \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " Jul 21 22:35:12.391231 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.391178 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-podinfo\") pod \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " Jul 21 22:35:12.391286 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.391255 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-envoy\") pod \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " Jul 21 22:35:12.391343 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.391285 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-data\") pod \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " Jul 21 22:35:12.391343 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.391290 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" (UID: "5531b6a1-3d90-4ed9-be46-ed8b43b7ea85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:35:12.391343 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.391323 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istiod-ca-cert\") pod \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " Jul 21 22:35:12.391545 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.391365 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-workload-socket\") pod \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " Jul 21 22:35:12.391545 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.391433 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-workload-certs\") pod \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\" (UID: \"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85\") " Jul 21 22:35:12.392104 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.391671 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" (UID: "5531b6a1-3d90-4ed9-be46-ed8b43b7ea85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:35:12.392104 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.391848 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istiod-ca-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:35:12.392104 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.391863 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-credential-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:35:12.392104 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.392048 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" (UID: "5531b6a1-3d90-4ed9-be46-ed8b43b7ea85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:35:12.392104 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.392070 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" (UID: "5531b6a1-3d90-4ed9-be46-ed8b43b7ea85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:35:12.392104 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.392080 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-data" (OuterVolumeSpecName: "istio-data") pod "5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" (UID: "5531b6a1-3d90-4ed9-be46-ed8b43b7ea85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:35:12.393510 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.393485 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-kube-api-access-kvx9w" (OuterVolumeSpecName: "kube-api-access-kvx9w") pod "5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" (UID: "5531b6a1-3d90-4ed9-be46-ed8b43b7ea85"). InnerVolumeSpecName "kube-api-access-kvx9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:35:12.393596 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.393564 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" (UID: "5531b6a1-3d90-4ed9-be46-ed8b43b7ea85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 22:35:12.393810 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.393789 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" (UID: "5531b6a1-3d90-4ed9-be46-ed8b43b7ea85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:35:12.393877 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.393863 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-token" (OuterVolumeSpecName: "istio-token") pod "5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" (UID: "5531b6a1-3d90-4ed9-be46-ed8b43b7ea85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:35:12.492905 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.492879 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-envoy\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:35:12.492905 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.492906 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-data\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:35:12.493049 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.492915 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-workload-socket\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:35:12.493049 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.492924 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-workload-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:35:12.493049 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.492932 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-token\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:35:12.493049 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.492940 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvx9w\" (UniqueName: \"kubernetes.io/projected/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-kube-api-access-kvx9w\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:35:12.493049 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.492948 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85-istio-podinfo\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\"" Jul 21 22:35:12.583934 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.583870 2580 generic.go:356] "Generic (PLEG): container finished" podID="5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" containerID="ea61c3520feab0e4291f0e47af8c156970d074cc31901cf642e1d26160c038c8" exitCode=0 Jul 21 22:35:12.583934 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.583925 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" Jul 21 22:35:12.584108 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.583962 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" event={"ID":"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85","Type":"ContainerDied","Data":"ea61c3520feab0e4291f0e47af8c156970d074cc31901cf642e1d26160c038c8"} Jul 21 22:35:12.584108 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.584029 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99" event={"ID":"5531b6a1-3d90-4ed9-be46-ed8b43b7ea85","Type":"ContainerDied","Data":"20e292f55adc20ffcad5b5928c6fb4a924cd4cee1ac31dcb9738cb08e6ad8694"} Jul 21 22:35:12.584108 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.584060 2580 scope.go:117] "RemoveContainer" containerID="ea61c3520feab0e4291f0e47af8c156970d074cc31901cf642e1d26160c038c8" Jul 21 22:35:12.595017 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.594998 2580 scope.go:117] "RemoveContainer" containerID="797c6ccbac57cb3b4fbb13ac5dccd6dd792647de67b902b49bc81aa53008b798" Jul 21 22:35:12.603226 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.603208 2580 scope.go:117] "RemoveContainer" containerID="ea61c3520feab0e4291f0e47af8c156970d074cc31901cf642e1d26160c038c8" Jul 21 22:35:12.603489 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:35:12.603466 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea61c3520feab0e4291f0e47af8c156970d074cc31901cf642e1d26160c038c8\": container with ID starting with ea61c3520feab0e4291f0e47af8c156970d074cc31901cf642e1d26160c038c8 not found: ID does not exist" containerID="ea61c3520feab0e4291f0e47af8c156970d074cc31901cf642e1d26160c038c8" Jul 21 22:35:12.603604 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.603505 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea61c3520feab0e4291f0e47af8c156970d074cc31901cf642e1d26160c038c8"} err="failed to get container status \"ea61c3520feab0e4291f0e47af8c156970d074cc31901cf642e1d26160c038c8\": rpc error: code = NotFound desc = could not find container \"ea61c3520feab0e4291f0e47af8c156970d074cc31901cf642e1d26160c038c8\": container with ID starting with ea61c3520feab0e4291f0e47af8c156970d074cc31901cf642e1d26160c038c8 not found: ID does not exist" Jul 21 22:35:12.603604 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.603529 2580 scope.go:117] "RemoveContainer" containerID="797c6ccbac57cb3b4fbb13ac5dccd6dd792647de67b902b49bc81aa53008b798" Jul 21 22:35:12.603776 ip-10-0-135-158 kubenswrapper[2580]: E0721 22:35:12.603758 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797c6ccbac57cb3b4fbb13ac5dccd6dd792647de67b902b49bc81aa53008b798\": container with ID starting with 797c6ccbac57cb3b4fbb13ac5dccd6dd792647de67b902b49bc81aa53008b798 not found: ID does not exist" containerID="797c6ccbac57cb3b4fbb13ac5dccd6dd792647de67b902b49bc81aa53008b798" Jul 21 22:35:12.603827 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.603781 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797c6ccbac57cb3b4fbb13ac5dccd6dd792647de67b902b49bc81aa53008b798"} err="failed to get container status \"797c6ccbac57cb3b4fbb13ac5dccd6dd792647de67b902b49bc81aa53008b798\": rpc error: code = NotFound desc = could not find container \"797c6ccbac57cb3b4fbb13ac5dccd6dd792647de67b902b49bc81aa53008b798\": container with ID starting with 797c6ccbac57cb3b4fbb13ac5dccd6dd792647de67b902b49bc81aa53008b798 not found: ID does not exist" Jul 21 22:35:12.608725 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.608701 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99"] Jul 21 22:35:12.612601 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.612583 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-b77603-openshift-default-8758fd895-sqx99"] Jul 21 22:35:12.783781 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:12.783754 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" path="/var/lib/kubelet/pods/5531b6a1-3d90-4ed9-be46-ed8b43b7ea85/volumes" Jul 21 22:35:13.216119 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:13.216082 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-rml2h_634a3cd8-16b4-417b-82ec-1f63d8c65dd7/manager/0.log" Jul 21 22:35:13.472513 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:13.472414 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55bc4db677-q2m5g_f26946dd-d05a-445e-878a-a96b65f407eb/manager/0.log" Jul 21 22:35:14.079003 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:14.078970 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54667d85dc-lwv8p_84aa0565-abdc-41bc-a77b-29f4f022854e/copy-manifests/0.log" Jul 21 22:35:14.087897 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:14.087872 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54667d85dc-lwv8p_84aa0565-abdc-41bc-a77b-29f4f022854e/manager/0.log" Jul 21 22:35:14.916736 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:14.916702 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-m7ls7_9b2239bf-33cf-4f63-a40d-92394cb9fde8/postgres/0.log" Jul 21 22:35:16.705175 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:16.705136 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-j69kr_fb55a8dc-fe35-44ae-9a0e-4b9178401a85/registry-server/0.log" Jul 21 22:35:16.829156 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:16.829120 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-d2p49_1b1ddb27-37c1-4f88-907f-bc569c6c971e/manager/0.log" Jul 21 22:35:17.453257 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:17.453227 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vpjh7_d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b/istio-proxy/0.log" Jul 21 22:35:17.941099 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:17.941071 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bwxc6_de321670-e3b1-4976-9799-1bcb1e8d4823/istio-proxy/1.log" Jul 21 22:35:18.069946 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:18.069905 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-h29c8_50b4ffb7-f0c9-4cc2-a090-ccbafe863a91/payload-pre-processing/0.log" Jul 21 22:35:18.191877 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:18.191792 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-pv4cl_ea66ae8c-0078-4574-80a9-0ae2dddec6ca/payload-processing/0.log" Jul 21 22:35:18.674120 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:18.674092 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-c7j4r_800d232d-d480-472a-8d5b-4f2d2290007d/main/0.log" Jul 21 22:35:19.152210 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:19.152184 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-5rc4b_d3918073-933b-4335-9171-1c070291e21c/main/0.log" Jul 21 22:35:27.765492 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:27.765456 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fb457_b3de97b9-626a-4516-9749-94ff62f60273/global-pull-secret-syncer/0.log" Jul 21 22:35:28.002750 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:28.002717 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wjhtb_0c0ed0aa-b6e4-4b63-8ab3-378c8e1024aa/konnectivity-agent/0.log" Jul 21 22:35:28.051970 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:28.051896 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-158.ec2.internal_9ba3d3cdb4f4f23180aa70b807d11a56/haproxy/0.log" Jul 21 22:35:32.597800 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:32.597771 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-j69kr_fb55a8dc-fe35-44ae-9a0e-4b9178401a85/registry-server/0.log" Jul 21 22:35:32.705589 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:32.705507 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-d2p49_1b1ddb27-37c1-4f88-907f-bc569c6c971e/manager/0.log" Jul 21 22:35:34.631277 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:34.631247 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f8ecf16-5c8e-48e1-bebc-cc747db6dd91/alertmanager/0.log" Jul 21 22:35:34.655916 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:34.655878 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f8ecf16-5c8e-48e1-bebc-cc747db6dd91/config-reloader/0.log" Jul 21 22:35:34.678914 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:34.678894 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f8ecf16-5c8e-48e1-bebc-cc747db6dd91/kube-rbac-proxy-web/0.log" Jul 21 22:35:34.702632 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:34.702611 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f8ecf16-5c8e-48e1-bebc-cc747db6dd91/kube-rbac-proxy/0.log" Jul 21 22:35:34.729564 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:34.729539 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f8ecf16-5c8e-48e1-bebc-cc747db6dd91/kube-rbac-proxy-metric/0.log" Jul 21 22:35:34.758506 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:34.758487 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f8ecf16-5c8e-48e1-bebc-cc747db6dd91/prom-label-proxy/0.log" Jul 21 22:35:34.784144 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:34.784125 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f8ecf16-5c8e-48e1-bebc-cc747db6dd91/init-config-reloader/0.log" Jul 21 22:35:34.862523 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:34.862454 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-jtbpf_3a164408-f1fb-46ab-9ae2-54216f57825e/kube-state-metrics/0.log" Jul 21 22:35:34.892221 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:34.892202 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-jtbpf_3a164408-f1fb-46ab-9ae2-54216f57825e/kube-rbac-proxy-main/0.log" Jul 21 22:35:34.917747 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:34.917733 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-jtbpf_3a164408-f1fb-46ab-9ae2-54216f57825e/kube-rbac-proxy-self/0.log" Jul 21 22:35:34.962418 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:34.962395 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-8645679d69-f7ndk_19851a0f-4eac-44f8-b73c-dfbc747236cf/metrics-server/0.log" Jul 21 22:35:35.002972 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.002948 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-tgl5d_cf08e0a6-5fee-4612-9805-3c91164df632/monitoring-plugin/0.log" Jul 21 22:35:35.051761 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.051727 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ct72r_908476c2-8321-46b7-b5b2-bc0d833e232c/node-exporter/0.log" Jul 21 22:35:35.088239 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.088205 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ct72r_908476c2-8321-46b7-b5b2-bc0d833e232c/kube-rbac-proxy/0.log" Jul 21 22:35:35.107282 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.107261 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ct72r_908476c2-8321-46b7-b5b2-bc0d833e232c/init-textfile/0.log" Jul 21 22:35:35.310308 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.310268 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-fz57t_b7b0ae59-42f4-4d6f-a389-a23f01a1e718/kube-rbac-proxy-main/0.log" Jul 21 22:35:35.333358 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.333332 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-fz57t_b7b0ae59-42f4-4d6f-a389-a23f01a1e718/kube-rbac-proxy-self/0.log" Jul 21 22:35:35.357150 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.357131 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-fz57t_b7b0ae59-42f4-4d6f-a389-a23f01a1e718/openshift-state-metrics/0.log" Jul 21 22:35:35.399074 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.399048 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec671505-9cec-4be4-9db2-a51589875186/prometheus/0.log" Jul 21 22:35:35.425034 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.425008 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec671505-9cec-4be4-9db2-a51589875186/config-reloader/0.log" Jul 21 22:35:35.449190 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.449173 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec671505-9cec-4be4-9db2-a51589875186/thanos-sidecar/0.log" Jul 21 22:35:35.475130 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.475109 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec671505-9cec-4be4-9db2-a51589875186/kube-rbac-proxy-web/0.log" Jul 21 22:35:35.503785 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.503766 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec671505-9cec-4be4-9db2-a51589875186/kube-rbac-proxy/0.log" Jul 21 22:35:35.530837 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.530820 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec671505-9cec-4be4-9db2-a51589875186/kube-rbac-proxy-thanos/0.log" Jul 21 22:35:35.563328 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.563276 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec671505-9cec-4be4-9db2-a51589875186/init-config-reloader/0.log" Jul 21 22:35:35.593343 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.593320 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-bspgk_148d42ef-cb3e-4912-bea7-942efecb697a/prometheus-operator/0.log" Jul 21 22:35:35.614262 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.614244 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-bspgk_148d42ef-cb3e-4912-bea7-942efecb697a/kube-rbac-proxy/0.log" Jul 21 22:35:35.641145 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.641127 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-jhvdw_80b73dfc-bb0f-4fc1-ae01-069daef10f00/prometheus-operator-admission-webhook/0.log" Jul 21 22:35:35.677256 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.677239 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-554d488d7f-77c9d_94e868f0-a955-4150-a07a-0874828f0ced/telemeter-client/0.log" Jul 21 22:35:35.702559 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.702535 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-554d488d7f-77c9d_94e868f0-a955-4150-a07a-0874828f0ced/reload/0.log" Jul 21 22:35:35.728319 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.728302 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-554d488d7f-77c9d_94e868f0-a955-4150-a07a-0874828f0ced/kube-rbac-proxy/0.log" Jul 21 22:35:35.772872 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.772852 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-644856656-r8znz_860ea5df-15b1-473d-a82b-fb8b89136cee/thanos-query/0.log" Jul 21 22:35:35.801415 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.801387 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-644856656-r8znz_860ea5df-15b1-473d-a82b-fb8b89136cee/kube-rbac-proxy-web/0.log" Jul 21 22:35:35.826303 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.826212 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-644856656-r8znz_860ea5df-15b1-473d-a82b-fb8b89136cee/kube-rbac-proxy/0.log" Jul 21 22:35:35.851059 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.851026 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-644856656-r8znz_860ea5df-15b1-473d-a82b-fb8b89136cee/prom-label-proxy/0.log" Jul 21 22:35:35.874987 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.874969 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-644856656-r8znz_860ea5df-15b1-473d-a82b-fb8b89136cee/kube-rbac-proxy-rules/0.log" Jul 21 22:35:35.901388 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:35.901359 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-644856656-r8znz_860ea5df-15b1-473d-a82b-fb8b89136cee/kube-rbac-proxy-metrics/0.log" Jul 21 22:35:38.058791 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.058755 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt"] Jul 21 22:35:38.059192 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.059179 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" containerName="istio-proxy" Jul 21 22:35:38.059234 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.059194 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" containerName="istio-proxy" Jul 21 22:35:38.059234 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.059212 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb92da7-67c6-449f-81a3-e5c53fd73e69" containerName="istio-proxy" Jul 21 22:35:38.059234 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.059217 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb92da7-67c6-449f-81a3-e5c53fd73e69" containerName="istio-proxy" Jul 21 22:35:38.059234 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.059230 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff10763a-6e60-4318-8c9f-95eb4565f372" containerName="main" Jul 21 22:35:38.059234 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.059236 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff10763a-6e60-4318-8c9f-95eb4565f372" containerName="main" Jul 21 22:35:38.059424 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.059247 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb92da7-67c6-449f-81a3-e5c53fd73e69" containerName="istio-proxy" Jul 21 22:35:38.059424 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.059253 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb92da7-67c6-449f-81a3-e5c53fd73e69" containerName="istio-proxy" Jul 21 22:35:38.059424 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.059315 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5531b6a1-3d90-4ed9-be46-ed8b43b7ea85" containerName="istio-proxy" Jul 21 22:35:38.059424 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.059321 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeb92da7-67c6-449f-81a3-e5c53fd73e69" containerName="istio-proxy" Jul 21 22:35:38.059424 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.059327 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff10763a-6e60-4318-8c9f-95eb4565f372" containerName="main" Jul 21 22:35:38.062640 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.062625 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:38.065490 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.065467 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wffls\"/\"default-dockercfg-fk87s\"" Jul 21 22:35:38.065609 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.065577 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wffls\"/\"openshift-service-ca.crt\"" Jul 21 22:35:38.066417 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.066403 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wffls\"/\"kube-root-ca.crt\"" Jul 21 22:35:38.074654 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.073972 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt"] Jul 21 22:35:38.126582 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.126554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/53c722d4-98d5-4f67-989a-dc8454e84aea-podres\") pod \"perf-node-gather-daemonset-zsdjt\" (UID: \"53c722d4-98d5-4f67-989a-dc8454e84aea\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:38.126699 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.126590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53c722d4-98d5-4f67-989a-dc8454e84aea-lib-modules\") pod \"perf-node-gather-daemonset-zsdjt\" (UID: \"53c722d4-98d5-4f67-989a-dc8454e84aea\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:38.126699 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.126609 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53c722d4-98d5-4f67-989a-dc8454e84aea-sys\") pod \"perf-node-gather-daemonset-zsdjt\" (UID: \"53c722d4-98d5-4f67-989a-dc8454e84aea\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:38.126699 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.126670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/53c722d4-98d5-4f67-989a-dc8454e84aea-proc\") pod \"perf-node-gather-daemonset-zsdjt\" (UID: \"53c722d4-98d5-4f67-989a-dc8454e84aea\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:38.126824 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.126783 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6cc2\" (UniqueName: \"kubernetes.io/projected/53c722d4-98d5-4f67-989a-dc8454e84aea-kube-api-access-b6cc2\") pod \"perf-node-gather-daemonset-zsdjt\" (UID: \"53c722d4-98d5-4f67-989a-dc8454e84aea\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:38.228016 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.227946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53c722d4-98d5-4f67-989a-dc8454e84aea-lib-modules\") pod \"perf-node-gather-daemonset-zsdjt\" (UID: \"53c722d4-98d5-4f67-989a-dc8454e84aea\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:38.228016 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.227977 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53c722d4-98d5-4f67-989a-dc8454e84aea-sys\") pod \"perf-node-gather-daemonset-zsdjt\" (UID: \"53c722d4-98d5-4f67-989a-dc8454e84aea\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:38.228016 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.228005 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/53c722d4-98d5-4f67-989a-dc8454e84aea-proc\") pod \"perf-node-gather-daemonset-zsdjt\" (UID: \"53c722d4-98d5-4f67-989a-dc8454e84aea\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:38.228253 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.228048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6cc2\" (UniqueName: \"kubernetes.io/projected/53c722d4-98d5-4f67-989a-dc8454e84aea-kube-api-access-b6cc2\") pod \"perf-node-gather-daemonset-zsdjt\" (UID: \"53c722d4-98d5-4f67-989a-dc8454e84aea\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:38.228253 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.228066 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53c722d4-98d5-4f67-989a-dc8454e84aea-sys\") pod \"perf-node-gather-daemonset-zsdjt\" (UID: \"53c722d4-98d5-4f67-989a-dc8454e84aea\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:38.228253 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.228090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/53c722d4-98d5-4f67-989a-dc8454e84aea-podres\") pod \"perf-node-gather-daemonset-zsdjt\" (UID: \"53c722d4-98d5-4f67-989a-dc8454e84aea\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:38.228253 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.228105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53c722d4-98d5-4f67-989a-dc8454e84aea-lib-modules\") pod \"perf-node-gather-daemonset-zsdjt\" (UID: \"53c722d4-98d5-4f67-989a-dc8454e84aea\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:38.228253 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.228117 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/53c722d4-98d5-4f67-989a-dc8454e84aea-proc\") pod \"perf-node-gather-daemonset-zsdjt\" (UID: \"53c722d4-98d5-4f67-989a-dc8454e84aea\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:38.228253 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.228185 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/53c722d4-98d5-4f67-989a-dc8454e84aea-podres\") pod \"perf-node-gather-daemonset-zsdjt\" (UID: \"53c722d4-98d5-4f67-989a-dc8454e84aea\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:38.236862 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.236842 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6cc2\" (UniqueName: \"kubernetes.io/projected/53c722d4-98d5-4f67-989a-dc8454e84aea-kube-api-access-b6cc2\") pod \"perf-node-gather-daemonset-zsdjt\" (UID: \"53c722d4-98d5-4f67-989a-dc8454e84aea\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:38.334171 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.334145 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7499db7949-q2wng_d9882f85-b8e1-45b9-8083-9cf6f9cf756c/console/0.log" Jul 21 22:35:38.373759 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.373733 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:38.505602 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.505576 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt"] Jul 21 22:35:38.507505 ip-10-0-135-158 kubenswrapper[2580]: W0721 22:35:38.507466 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod53c722d4_98d5_4f67_989a_dc8454e84aea.slice/crio-8e7299b7e9f879d92b5014958036496eb8310fd4ff89f2ae4d85ec23b12e6c25 WatchSource:0}: Error finding container 8e7299b7e9f879d92b5014958036496eb8310fd4ff89f2ae4d85ec23b12e6c25: Status 404 returned error can't find the container with id 8e7299b7e9f879d92b5014958036496eb8310fd4ff89f2ae4d85ec23b12e6c25 Jul 21 22:35:38.727292 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.727260 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" event={"ID":"53c722d4-98d5-4f67-989a-dc8454e84aea","Type":"ContainerStarted","Data":"a55c51ff91740faea047a59d0f417b6dd8a6265918b85026e99f68929fc9d713"} Jul 21 22:35:38.727501 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.727311 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" event={"ID":"53c722d4-98d5-4f67-989a-dc8454e84aea","Type":"ContainerStarted","Data":"8e7299b7e9f879d92b5014958036496eb8310fd4ff89f2ae4d85ec23b12e6c25"} Jul 21 22:35:38.727501 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.727409 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:38.747415 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:38.747354 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" podStartSLOduration=0.747340624 podStartE2EDuration="747.340624ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 22:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:35:38.746617527 +0000 UTC m=+2816.509851788" watchObservedRunningTime="2026-07-21 22:35:38.747340624 +0000 UTC m=+2816.510574885" Jul 21 22:35:40.130340 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:40.130295 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5l4xt_8a67aeca-9eb3-48be-89b6-27696cefec9d/dns/0.log" Jul 21 22:35:40.157720 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:40.157696 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5l4xt_8a67aeca-9eb3-48be-89b6-27696cefec9d/kube-rbac-proxy/0.log" Jul 21 22:35:40.315714 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:40.315681 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fk4dn_5daa5430-2084-48d2-a008-6305a894ea65/dns-node-resolver/0.log" Jul 21 22:35:41.006125 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:41.006098 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jdcql_d6c5658d-ed28-4bef-ada7-ab4746a7a346/node-ca/0.log" Jul 21 22:35:42.028444 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:42.028419 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vpjh7_d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b/istio-proxy/0.log" Jul 21 22:35:42.337159 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:42.337074 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bwxc6_de321670-e3b1-4976-9799-1bcb1e8d4823/istio-proxy/0.log" Jul 21 22:35:42.361919 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:42.361898 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bwxc6_de321670-e3b1-4976-9799-1bcb1e8d4823/istio-proxy/1.log" Jul 21 22:35:42.396632 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:42.396612 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-h29c8_50b4ffb7-f0c9-4cc2-a090-ccbafe863a91/payload-pre-processing/0.log" Jul 21 22:35:42.432240 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:42.431729 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-pv4cl_ea66ae8c-0078-4574-80a9-0ae2dddec6ca/payload-processing/0.log" Jul 21 22:35:43.244070 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:43.244045 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hdhhq_529698f9-3ae6-4d6a-a0dc-839a22bc5119/serve-healthcheck-canary/0.log" Jul 21 22:35:43.771700 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:43.771660 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pdv8n_6eb0fb30-a50c-4913-b8e0-907e9fc18c7a/kube-rbac-proxy/0.log" Jul 21 22:35:43.795534 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:43.795511 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pdv8n_6eb0fb30-a50c-4913-b8e0-907e9fc18c7a/exporter/0.log" Jul 21 22:35:43.819535 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:43.819513 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pdv8n_6eb0fb30-a50c-4913-b8e0-907e9fc18c7a/extractor/0.log" Jul 21 22:35:44.743119 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:44.743090 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-zsdjt" Jul 21 22:35:46.112901 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:46.112870 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-rml2h_634a3cd8-16b4-417b-82ec-1f63d8c65dd7/manager/0.log" Jul 21 22:35:46.550928 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:46.550894 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55bc4db677-q2m5g_f26946dd-d05a-445e-878a-a96b65f407eb/manager/0.log" Jul 21 22:35:46.885035 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:46.884992 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54667d85dc-lwv8p_84aa0565-abdc-41bc-a77b-29f4f022854e/manager/0.log" Jul 21 22:35:46.910945 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:46.910926 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54667d85dc-lwv8p_84aa0565-abdc-41bc-a77b-29f4f022854e/copy-manifests/0.log" Jul 21 22:35:48.365738 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:48.365677 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-d89c66ff-94xzl_6f69b996-0878-41ef-9a1c-d7d1726e64fb/manager/0.log" Jul 21 22:35:48.406562 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:48.406537 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vpjh7_d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b/istio-proxy/0.log" Jul 21 22:35:48.420225 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:48.420170 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-9kh97_db2829c9-411b-4d5a-af2d-0c3c753ff36a/openshift-lws-operator/0.log" Jul 21 22:35:48.673923 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:48.673846 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bwxc6_de321670-e3b1-4976-9799-1bcb1e8d4823/istio-proxy/0.log" Jul 21 22:35:48.699877 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:48.699852 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bwxc6_de321670-e3b1-4976-9799-1bcb1e8d4823/istio-proxy/1.log" Jul 21 22:35:48.720024 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:48.719998 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-h29c8_50b4ffb7-f0c9-4cc2-a090-ccbafe863a91/payload-pre-processing/0.log" Jul 21 22:35:48.738244 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:48.738226 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-pv4cl_ea66ae8c-0078-4574-80a9-0ae2dddec6ca/payload-processing/0.log" Jul 21 22:35:49.953574 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:49.953541 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vpjh7_d557bb12-34d9-4b8b-bd9f-e8b1cf152e1b/istio-proxy/0.log" Jul 21 22:35:50.278987 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:50.278956 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bwxc6_de321670-e3b1-4976-9799-1bcb1e8d4823/istio-proxy/0.log" Jul 21 22:35:50.303874 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:50.303848 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bwxc6_de321670-e3b1-4976-9799-1bcb1e8d4823/istio-proxy/1.log" Jul 21 22:35:50.329314 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:50.329282 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-h29c8_50b4ffb7-f0c9-4cc2-a090-ccbafe863a91/payload-pre-processing/0.log" Jul 21 22:35:50.352206 ip-10-0-135-158 kubenswrapper[2580]: I0721 22:35:50.352186 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-pv4cl_ea66ae8c-0078-4574-80a9-0ae2dddec6ca/payload-processing/0.log"