Jul 06 23:24:49.170989 ip-10-0-135-248 systemd[1]: Starting Kubernetes Kubelet... Jul 06 23:24:49.544910 ip-10-0-135-248 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 23:24:49.544910 ip-10-0-135-248 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 23:24:49.544910 ip-10-0-135-248 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 23:24:49.544910 ip-10-0-135-248 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 23:24:49.544910 ip-10-0-135-248 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 23:24:49.546478 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.546339 2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 23:24:49.551794 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551770 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:24:49.551794 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551789 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:24:49.551794 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551793 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:24:49.551794 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551797 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:24:49.551794 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551800 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:24:49.551794 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551803 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551806 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551808 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551811 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551814 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551816 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551819 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551821 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551824 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551826 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551829 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551831 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551834 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551837 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551839 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551845 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551848 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551851 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551854 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551857 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:24:49.552022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551859 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551861 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551864 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551866 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551869 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551871 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551874 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551876 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551879 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551881 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551884 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551886 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551889 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551892 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551895 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551897 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551900 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551902 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551905 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551908 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:24:49.552509 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551910 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551913 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551915 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551918 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551920 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551923 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551926 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551929 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551932 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551934 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551937 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551939 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551943 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551948 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551951 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551954 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551957 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551960 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551963 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:24:49.553022 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551966 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551970 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551972 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551975 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551977 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551981 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551986 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551989 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551992 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551995 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.551997 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552000 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552002 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552005 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552008 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552010 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552013 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552015 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552018 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:24:49.553497 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552021 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552024 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552027 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552030 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552033 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552036 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552038 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552040 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552043 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552045 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552048 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552051 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552053 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552056 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552058 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552060 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552773 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552780 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552782 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552785 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:24:49.553963 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552789 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552792 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552795 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552797 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552800 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552803 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552805 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552808 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552810 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552813 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552816 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552818 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552821 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552824 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552826 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552829 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552831 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552834 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552836 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552839 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:24:49.554480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552841 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552844 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552846 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552849 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552852 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552854 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552857 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552863 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552867 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552870 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552873 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552875 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552878 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552881 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552884 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552886 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552889 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552891 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552894 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:24:49.554993 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552896 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552899 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552901 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552904 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552907 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552909 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552912 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552915 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552917 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552920 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552922 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552925 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552927 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552930 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552932 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552935 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552938 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552940 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552943 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552945 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:24:49.555480 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552948 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552950 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552953 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552955 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552957 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552960 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552963 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552966 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552968 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552972 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552975 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552978 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552981 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552983 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552986 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552988 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552990 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552993 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.552997 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:24:49.555985 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553000 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553004 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553007 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553009 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553012 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553014 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553017 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553019 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553022 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553025 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553028 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553030 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553033 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553035 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553038 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553040 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553043 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553404 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553411 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553414 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:24:49.556469 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553416 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553419 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553422 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553425 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553429 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553432 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553435 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553438 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553440 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553443 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553446 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553448 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553450 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553453 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553458 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553460 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553464 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553467 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553473 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:24:49.556975 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553476 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553479 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553481 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553484 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553486 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553489 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553491 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553494 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553496 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553499 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553502 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553506 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553508 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553511 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553514 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553517 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553519 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553521 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553524 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553526 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:24:49.557421 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553530 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553534 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553537 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553539 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553542 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553545 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553547 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553550 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553553 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553556 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553559 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553561 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553564 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553566 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553569 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553571 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553573 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553576 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553579 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:24:49.557939 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553581 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553583 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553586 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553588 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553592 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553594 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553597 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553600 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553604 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553607 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553609 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553611 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553614 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553617 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553619 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553622 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553624 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553627 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553629 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553631 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:24:49.558388 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553634 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553638 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553640 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553643 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553646 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553648 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553650 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553653 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553656 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553658 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553660 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553663 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553665 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553668 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553670 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553672 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553676 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.553678 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553771 2571 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553779 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553786 2571 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 23:24:49.558893 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553790 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553794 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553797 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553801 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553804 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553808 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553811 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553814 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553817 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553820 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553823 2571 flags.go:64] FLAG: --cgroup-root="" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553827 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553829 2571 flags.go:64] FLAG: --client-ca-file="" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553833 2571 flags.go:64] FLAG: --cloud-provider="external" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553836 2571 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553840 2571 flags.go:64] FLAG: --cluster-domain="" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553843 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553845 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553849 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553852 2571 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553855 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553858 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553861 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 23:24:49.559400 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553864 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553867 2571 flags.go:64] FLAG: --contention-profiling="false" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553869 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553872 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553876 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553879 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553883 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553886 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553889 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553892 2571 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553894 2571 flags.go:64] FLAG: --enable-server="true" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553897 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553901 2571 flags.go:64] FLAG: --event-burst="100" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553904 2571 flags.go:64] FLAG: --event-qps="50" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553907 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553910 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553912 2571 flags.go:64] FLAG: --eviction-hard="" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553916 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553919 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553922 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553924 2571 flags.go:64] FLAG: --eviction-soft="" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553928 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553931 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553934 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553937 2571 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 23:24:49.559957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553939 2571 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553942 2571 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553945 2571 flags.go:64] FLAG: --feature-gates="" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553948 2571 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553951 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553954 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553957 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553960 2571 flags.go:64] FLAG: --healthz-port="10248" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553962 2571 flags.go:64] FLAG: --help="false" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553965 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-135-248.ec2.internal" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553969 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553973 2571 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553976 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553979 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553983 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553985 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553988 2571 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553990 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553993 2571 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553996 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.553998 2571 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554001 2571 flags.go:64] FLAG: --kube-reserved="" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554004 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554006 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 23:24:49.560630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554009 2571 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554012 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554016 2571 flags.go:64] FLAG: --lock-file="" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554019 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554021 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554024 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554029 2571 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554032 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554035 2571 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554038 2571 flags.go:64] FLAG: --logging-format="text" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554041 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554044 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554046 2571 flags.go:64] FLAG: --manifest-url="" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554049 2571 flags.go:64] FLAG: --manifest-url-header="" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554053 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554056 2571 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554059 2571 flags.go:64] FLAG: --max-pods="110" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554062 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554065 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554068 2571 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554071 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554074 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554077 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554079 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554091 2571 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 23:24:49.561235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554094 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554097 2571 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554100 2571 flags.go:64] FLAG: --pod-cidr="" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554102 2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554108 2571 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554110 2571 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554113 2571 flags.go:64] FLAG: --pods-per-core="0" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554116 2571 flags.go:64] FLAG: --port="10250" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554119 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554123 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04e99880ba1667d00" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554126 2571 flags.go:64] FLAG: --qos-reserved="" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554129 2571 flags.go:64] FLAG: --read-only-port="10255" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554131 2571 flags.go:64] FLAG: --register-node="true" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554134 2571 flags.go:64] FLAG: --register-with-taints="" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554138 2571 flags.go:64] FLAG: --registry-burst="10" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554141 2571 flags.go:64] FLAG: --registry-qps="5" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554143 2571 flags.go:64] FLAG: --reserved-cpus="" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554146 2571 flags.go:64] FLAG: --reserved-memory="" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554149 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554152 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554155 2571 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554158 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554160 2571 flags.go:64] FLAG: --runonce="false" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554163 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554165 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 23:24:49.561854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554168 2571 flags.go:64] FLAG: --seccomp-default="false" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554171 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554174 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554177 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554180 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554183 2571 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554186 2571 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554188 2571 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554191 2571 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554193 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554196 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554199 2571 flags.go:64] FLAG: --system-cgroups="" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554202 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554207 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554209 2571 flags.go:64] FLAG: --tls-cert-file="" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554212 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554217 2571 flags.go:64] FLAG: --tls-min-version="" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554220 2571 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554222 2571 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554225 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554228 2571 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554231 2571 flags.go:64] FLAG: --v="2" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554234 2571 flags.go:64] FLAG: --version="false" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554239 2571 flags.go:64] FLAG: --vmodule="" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554242 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 23:24:49.562460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554245 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554356 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554360 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554363 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554366 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554369 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554372 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554375 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554377 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554380 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554383 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554386 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554388 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554391 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554394 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554396 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554399 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554402 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554404 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554406 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:24:49.563138 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554409 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554412 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554414 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554418 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554420 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554423 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554425 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554428 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554431 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554434 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554437 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554440 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554443 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554445 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554448 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554451 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554453 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554456 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554458 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554460 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:24:49.563621 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554463 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554465 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554468 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554470 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554473 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554475 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554478 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554480 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554483 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554486 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554489 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554491 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554494 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554496 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554499 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554502 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554505 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554507 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554510 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:24:49.564213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554512 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554514 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554517 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554519 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554521 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554524 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554527 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554529 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554532 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554534 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554536 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554539 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554542 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554544 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554547 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554550 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554552 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554555 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554558 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554560 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:24:49.564701 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554563 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554565 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554568 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554570 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554573 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554575 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554579 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554583 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554588 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554590 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554593 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554596 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554599 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554603 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554606 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554609 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554612 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554614 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554618 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:24:49.565214 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554620 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:24:49.565673 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.554623 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:24:49.565673 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.554629 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 23:24:49.565673 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.561596 2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 23:24:49.565673 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.561610 2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 23:24:49.565673 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.561629 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 23:24:49.565673 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.561634 2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 23:24:49.565673 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561687 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:24:49.565673 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561693 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:24:49.565673 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561696 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:24:49.565673 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561699 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:24:49.565673 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561702 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:24:49.565673 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561704 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:24:49.565673 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561707 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:24:49.565673 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561710 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:24:49.565673 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561712 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561735 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561739 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561743 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561747 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561750 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561753 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561756 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561760 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561764 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561767 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561770 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561773 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561775 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561778 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561781 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561783 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561786 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561789 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:24:49.566063 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561791 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561794 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561797 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561800 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561802 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561805 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561807 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561810 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561812 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561815 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561817 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561819 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561822 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561825 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561827 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561829 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561832 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561835 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561837 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561840 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:24:49.566521 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561842 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561845 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561847 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561850 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561852 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561855 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561857 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561860 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561864 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561868 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561871 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561873 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561876 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561879 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561882 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561885 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561887 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561889 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561892 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561894 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:24:49.567090 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561897 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561899 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561902 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561904 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561907 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561909 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561911 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561914 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561916 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561919 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561921 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561924 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561926 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561929 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561931 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561934 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561936 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561938 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561941 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:24:49.567579 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561943 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:24:49.568097 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561946 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:24:49.568097 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561948 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:24:49.568097 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561950 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:24:49.568097 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561953 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:24:49.568097 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561955 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:24:49.568097 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561957 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:24:49.568097 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561960 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:24:49.568097 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561962 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:24:49.568097 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561966 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:24:49.568097 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561970 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:24:49.568097 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561973 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:24:49.568097 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.561976 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:24:49.568097 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.561981 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 23:24:49.568097 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562090 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562095 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562098 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562101 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562103 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562107 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562111 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562114 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562117 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562120 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562122 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562125 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562128 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562131 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562134 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562136 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562139 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562142 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562144 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562147 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 23:24:49.568450 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562149 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562152 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562155 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562157 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562159 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562162 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562164 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562167 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562173 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562176 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562178 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562182 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562184 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562187 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562189 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562192 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562194 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562197 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562199 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562201 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 23:24:49.568944 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562204 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562206 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562208 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562211 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562213 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562216 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562218 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562220 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562223 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562225 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562228 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562230 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562232 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562235 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562237 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562240 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562242 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562245 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562247 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562250 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 23:24:49.569413 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562252 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562255 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562260 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562262 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562266 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562269 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562272 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562275 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562277 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562280 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562282 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562286 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562289 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562292 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562294 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562297 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562299 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562302 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562304 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 23:24:49.569914 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562306 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562309 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562312 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562314 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562316 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562319 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562321 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562324 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562326 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562328 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562331 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562333 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562338 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562340 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562343 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562347 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562349 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562352 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562354 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 23:24:49.570366 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:49.562357 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 23:24:49.570875 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.562362 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 23:24:49.570875 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.562462 2571 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 23:24:49.570875 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.564483 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 23:24:49.570875 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.566328 2571 server.go:1019] "Starting client certificate rotation" Jul 06 23:24:49.570875 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.566430 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 23:24:49.570875 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.567194 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 23:24:49.570875 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.568500 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 23:24:49.576344 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.576321 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 23:24:49.585678 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.585662 2571 log.go:25] "Validated CRI v1 runtime API" Jul 06 23:24:49.587664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.587649 2571 log.go:25] "Validated CRI v1 image API" Jul 06 23:24:49.589879 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.589864 2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 23:24:49.592821 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.592797 2571 fs.go:135] Filesystem UUIDs: map[116fc1f4-6fc4-434a-8ffa-e62e65ee13ba:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ee99c17d-9bb3-4fe0-91c0-5189ee0b1b34:/dev/nvme0n1p3] Jul 06 23:24:49.592897 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.592820 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 23:24:49.595412 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.595397 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 23:24:49.597893 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.597786 2571 manager.go:216] Machine: {Timestamp:2026-07-06 23:24:49.596444059 +0000 UTC m=+0.310498608 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089714 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26fdfb4ecc5f0d4b3cb0420e3bbca4 SystemUUID:ec26fdfb-4ecc-5f0d-4b3c-b0420e3bbca4 BootID:2f8f5e10-e9d5-4344-814d-a8ca2f58465f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:67:f0:ed:64:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:67:f0:ed:64:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:a3:07:7b:44:e6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 23:24:49.597893 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.597892 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 23:24:49.597996 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.597973 2571 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 23:24:49.600496 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.600468 2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 23:24:49.600638 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.600500 2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-248.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 23:24:49.601137 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.601127 2571 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 23:24:49.601167 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.601139 2571 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 23:24:49.601167 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.601151 2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 23:24:49.601780 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.601769 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 23:24:49.603021 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.603011 2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 23:24:49.605734 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.605710 2571 state_mem.go:36] "Initialized new in-memory state store" Jul 06 23:24:49.605846 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.605836 2571 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 23:24:49.605880 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.605865 2571 kubelet.go:486] "Attempting to sync node with API server" Jul 06 23:24:49.605880 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.605875 2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 23:24:49.605936 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.605888 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 23:24:49.605936 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.605898 2571 kubelet.go:398] "Adding apiserver pod source" Jul 06 23:24:49.605936 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.605907 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 23:24:49.606249 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.606237 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 23:24:49.606298 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.606256 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 23:24:49.608157 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.608136 2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 23:24:49.609836 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.609824 2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 23:24:49.609901 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.609853 2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 23:24:49.610954 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.610942 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 23:24:49.611012 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.610958 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 23:24:49.611012 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.610964 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 23:24:49.611012 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.610970 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 23:24:49.611012 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.610979 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 23:24:49.611012 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.610984 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 23:24:49.611012 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.610992 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 23:24:49.611012 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.610997 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 23:24:49.611012 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.611004 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 23:24:49.611012 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.611008 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 23:24:49.611012 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.611016 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 23:24:49.611288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.611031 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 23:24:49.611288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.611056 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 23:24:49.611288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.611062 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 23:24:49.614408 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.614392 2571 server.go:1268] "Started kubelet" Jul 06 23:24:49.614527 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.614495 2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 23:24:49.618182 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.617926 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 23:24:49.618182 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.618015 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 23:24:49.618712 ip-10-0-135-248 systemd[1]: Started Kubernetes Kubelet. Jul 06 23:24:49.620411 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.619515 2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 23:24:49.620411 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.619967 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-248.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 23:24:49.620411 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:49.620255 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 23:24:49.620825 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.620696 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-npwxh" Jul 06 23:24:49.620825 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.620738 2571 server.go:310] "Adding debug handlers to kubelet server" Jul 06 23:24:49.624238 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:49.624218 2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 23:24:49.625755 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:49.625737 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 23:24:49.626247 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:49.625388 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-248.ec2.internal.18bfd6ecb5e5acf7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-248.ec2.internal,UID:ip-10-0-135-248.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-248.ec2.internal,},FirstTimestamp:2026-07-06 23:24:49.614359799 +0000 UTC m=+0.328414408,LastTimestamp:2026-07-06 23:24:49.614359799 +0000 UTC m=+0.328414408,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-248.ec2.internal,}" Jul 06 23:24:49.627403 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.627383 2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 23:24:49.627403 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.627398 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 23:24:49.627596 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:49.627575 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-248.ec2.internal\" not found" Jul 06 23:24:49.628578 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.628561 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 23:24:49.628667 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.628587 2571 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 23:24:49.628667 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.628608 2571 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 23:24:49.628780 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.628761 2571 reconstruct.go:97] "Volume reconstruction finished" Jul 06 23:24:49.630366 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.629494 2571 reconciler.go:29] "Reconciler: start to sync state" Jul 06 23:24:49.630575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.630546 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-npwxh" Jul 06 23:24:49.630705 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.630688 2571 factory.go:55] Registering systemd factory Jul 06 23:24:49.630705 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.630706 2571 factory.go:223] Registration of the systemd container factory successfully Jul 06 23:24:49.631151 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.631129 2571 factory.go:153] Registering CRI-O factory Jul 06 23:24:49.631151 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.631146 2571 factory.go:223] Registration of the crio container factory successfully Jul 06 23:24:49.631273 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.631234 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 23:24:49.631273 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:49.631241 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 23:24:49.631273 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.631260 2571 factory.go:103] Registering Raw factory Jul 06 23:24:49.631448 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:49.631247 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 23:24:49.631448 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.631293 2571 manager.go:1196] Started watching for new ooms in manager Jul 06 23:24:49.631874 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.631860 2571 manager.go:349] Starting recovery of all containers Jul 06 23:24:49.637537 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.637487 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 23:24:49.643946 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.643830 2571 manager.go:354] Recovery completed Jul 06 23:24:49.648119 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.648103 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:24:49.651120 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.651105 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:24:49.651208 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.651138 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:24:49.651208 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.651151 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:24:49.652149 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.652130 2571 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 23:24:49.652149 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.652149 2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 23:24:49.652287 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.652167 2571 state_mem.go:36] "Initialized new in-memory state store" Jul 06 23:24:49.652558 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.652544 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:24:49.653454 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.653314 2571 policy_none.go:49] "None policy: Start" Jul 06 23:24:49.653454 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.653333 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 23:24:49.653454 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.653348 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 23:24:49.653938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.653659 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:24:49.653995 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.653961 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:24:49.653995 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.653974 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:24:49.654569 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.654559 2571 policy_none.go:47] "Start" Jul 06 23:24:49.690245 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.690231 2571 manager.go:337] "Starting Device Plugin manager" Jul 06 23:24:49.690339 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:49.690261 2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 23:24:49.690339 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.690270 2571 server.go:87] "Starting device plugin registration server" Jul 06 23:24:49.690516 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.690504 2571 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 23:24:49.690565 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.690520 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 23:24:49.690752 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.690654 2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 23:24:49.690833 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.690773 2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 23:24:49.690833 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.690785 2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 23:24:49.691438 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:49.691418 2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 23:24:49.691514 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:49.691473 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-248.ec2.internal\" not found" Jul 06 23:24:49.779516 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.779498 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 23:24:49.780506 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.779533 2571 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 23:24:49.780506 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.779551 2571 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 23:24:49.780506 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:49.779577 2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 23:24:49.782103 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.782083 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:24:49.791263 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.791245 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:24:49.792013 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.791996 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:24:49.792095 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.792024 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:24:49.792095 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.792035 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:24:49.792095 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.792055 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-248.ec2.internal" Jul 06 23:24:49.798001 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.797966 2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-248.ec2.internal" Jul 06 23:24:49.798001 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:49.797986 2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-248.ec2.internal\": node \"ip-10-0-135-248.ec2.internal\" not found" Jul 06 23:24:49.810343 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:49.810325 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-248.ec2.internal\" not found" Jul 06 23:24:49.880313 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.880294 2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal"] Jul 06 23:24:49.880388 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.880352 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:24:49.881125 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.881111 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:24:49.881184 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.881135 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:24:49.881184 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.881145 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:24:49.883912 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.883901 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:24:49.884036 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.884015 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal" Jul 06 23:24:49.884070 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.884044 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:24:49.884627 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.884598 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:24:49.884627 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.884615 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:24:49.884627 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.884628 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:24:49.884831 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.884636 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:24:49.884831 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.884643 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:24:49.884831 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.884647 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:24:49.885679 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.885665 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal" Jul 06 23:24:49.885744 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.885689 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 23:24:49.886332 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.886320 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientMemory" Jul 06 23:24:49.886413 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.886376 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 23:24:49.886413 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.886392 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeHasSufficientPID" Jul 06 23:24:49.909241 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:49.909221 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-248.ec2.internal\" not found" node="ip-10-0-135-248.ec2.internal" Jul 06 23:24:49.911009 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:49.910996 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-248.ec2.internal\" not found" Jul 06 23:24:49.913573 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:49.913560 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-248.ec2.internal\" not found" node="ip-10-0-135-248.ec2.internal" Jul 06 23:24:49.930651 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.930637 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/84e3d258c0c669908efda319c624e11f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal\" (UID: \"84e3d258c0c669908efda319c624e11f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal" Jul 06 23:24:49.930735 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.930659 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84e3d258c0c669908efda319c624e11f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal\" (UID: \"84e3d258c0c669908efda319c624e11f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal" Jul 06 23:24:49.930735 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:49.930676 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7916f8702abca842d06320c25e14c3c0-config\") pod \"kube-apiserver-proxy-ip-10-0-135-248.ec2.internal\" (UID: \"7916f8702abca842d06320c25e14c3c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal" Jul 06 23:24:50.012048 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:50.012035 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-248.ec2.internal\" not found" Jul 06 23:24:50.031530 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.031502 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/84e3d258c0c669908efda319c624e11f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal\" (UID: \"84e3d258c0c669908efda319c624e11f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal" Jul 06 23:24:50.031625 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.031538 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84e3d258c0c669908efda319c624e11f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal\" (UID: \"84e3d258c0c669908efda319c624e11f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal" Jul 06 23:24:50.031625 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.031565 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7916f8702abca842d06320c25e14c3c0-config\") pod \"kube-apiserver-proxy-ip-10-0-135-248.ec2.internal\" (UID: \"7916f8702abca842d06320c25e14c3c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal" Jul 06 23:24:50.031625 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.031615 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/84e3d258c0c669908efda319c624e11f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal\" (UID: \"84e3d258c0c669908efda319c624e11f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal" Jul 06 23:24:50.031760 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.031626 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84e3d258c0c669908efda319c624e11f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal\" (UID: \"84e3d258c0c669908efda319c624e11f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal" Jul 06 23:24:50.031760 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.031615 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7916f8702abca842d06320c25e14c3c0-config\") pod \"kube-apiserver-proxy-ip-10-0-135-248.ec2.internal\" (UID: \"7916f8702abca842d06320c25e14c3c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal" Jul 06 23:24:50.112857 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:50.112820 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-248.ec2.internal\" not found" Jul 06 23:24:50.211324 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.211301 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal" Jul 06 23:24:50.213389 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:50.213376 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-248.ec2.internal\" not found" Jul 06 23:24:50.215995 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.215979 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal" Jul 06 23:24:50.314433 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:50.314413 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-248.ec2.internal\" not found" Jul 06 23:24:50.414944 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:50.414922 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-248.ec2.internal\" not found" Jul 06 23:24:50.434871 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.434852 2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:24:50.527871 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.527851 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal" Jul 06 23:24:50.538551 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.538537 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 23:24:50.540265 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.540250 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal" Jul 06 23:24:50.547192 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.547172 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 23:24:50.565265 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.565250 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 23:24:50.565375 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.565357 2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 23:24:50.565414 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.565386 2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 23:24:50.606581 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.606560 2571 apiserver.go:52] "Watching apiserver" Jul 06 23:24:50.623051 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.621771 2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 23:24:50.623051 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.622608 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-bgdhp","openshift-multus/network-metrics-daemon-mdnxn","openshift-network-diagnostics/network-check-target-v8xhl","kube-system/global-pull-secret-syncer-rsnfg","kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal","openshift-dns/node-resolver-l6dhh","openshift-image-registry/node-ca-kpwsm","openshift-network-operator/iptables-alerter-fwj6n","openshift-ovn-kubernetes/ovnkube-node-c7gg2","kube-system/konnectivity-agent-jdmhg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6","openshift-cluster-node-tuning-operator/tuned-4bk8b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal","openshift-multus/multus-additional-cni-plugins-bvl5t"] Jul 06 23:24:50.624311 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.624284 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.625327 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.625309 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:24:50.625396 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:50.625375 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdnxn" podUID="f9260140-57c2-4913-adc3-7497f54cd33b" Jul 06 23:24:50.626440 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.626424 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 23:24:50.626508 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.626492 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 23:24:50.626561 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.626431 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:24:50.626610 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:50.626587 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:24:50.626652 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.626606 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rnk28\"" Jul 06 23:24:50.626745 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.626734 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 23:24:50.626850 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.626836 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 23:24:50.627471 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.627458 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 23:24:50.628530 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.628515 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:24:50.628599 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:50.628563 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsnfg" podUID="a58729bb-f5a9-4bce-8cfc-3f98f6dd2401" Jul 06 23:24:50.629933 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.629920 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l6dhh" Jul 06 23:24:50.630962 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.630943 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kpwsm" Jul 06 23:24:50.631126 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.631105 2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:24:50.631813 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.631795 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 23:24:50.631894 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.631839 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-52jmk\"" Jul 06 23:24:50.632091 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.632077 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fwj6n" Jul 06 23:24:50.632171 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.632134 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 23:24:50.632915 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.632898 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 23:24:50.633023 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.633007 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 23:24:50.633068 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.633053 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 23:24:50.633388 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.633217 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2brkp\"" Jul 06 23:24:50.633388 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.633316 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.634000 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.633961 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkdtp\" (UniqueName: \"kubernetes.io/projected/e746a856-21ba-4c28-9545-6e6d6e299f49-kube-api-access-mkdtp\") pod \"node-resolver-l6dhh\" (UID: \"e746a856-21ba-4c28-9545-6e6d6e299f49\") " pod="openshift-dns/node-resolver-l6dhh" Jul 06 23:24:50.634057 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634017 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-cni-binary-copy\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.634057 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634049 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-hostroot\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.634126 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634083 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e746a856-21ba-4c28-9545-6e6d6e299f49-tmp-dir\") pod \"node-resolver-l6dhh\" (UID: \"e746a856-21ba-4c28-9545-6e6d6e299f49\") " pod="openshift-dns/node-resolver-l6dhh" Jul 06 23:24:50.634183 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634158 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-host-run-k8s-cni-cncf-io\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.634234 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634196 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-host-run-netns\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.634273 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634256 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-host-var-lib-cni-multus\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.634316 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634302 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7xc5\" (UniqueName: \"kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5\") pod \"network-check-target-v8xhl\" (UID: \"8125ec7d-1a89-4270-af40-9b10a7f44cf5\") " pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:24:50.634355 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634346 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-kubelet-config\") pod \"global-pull-secret-syncer-rsnfg\" (UID: \"a58729bb-f5a9-4bce-8cfc-3f98f6dd2401\") " pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:24:50.634408 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634371 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-host-var-lib-kubelet\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.634457 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634410 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hgnhz\"" Jul 06 23:24:50.634457 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634412 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jk5j\" (UniqueName: \"kubernetes.io/projected/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-kube-api-access-7jk5j\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.634457 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634438 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:24:50.634577 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634387 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 23:24:50.634577 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634469 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvspz\" (UniqueName: \"kubernetes.io/projected/f9260140-57c2-4913-adc3-7497f54cd33b-kube-api-access-jvspz\") pod \"network-metrics-daemon-mdnxn\" (UID: \"f9260140-57c2-4913-adc3-7497f54cd33b\") " pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:24:50.634577 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634499 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-dbus\") pod \"global-pull-secret-syncer-rsnfg\" (UID: \"a58729bb-f5a9-4bce-8cfc-3f98f6dd2401\") " pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:24:50.634577 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634533 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret\") pod \"global-pull-secret-syncer-rsnfg\" (UID: \"a58729bb-f5a9-4bce-8cfc-3f98f6dd2401\") " pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:24:50.634577 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634565 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e746a856-21ba-4c28-9545-6e6d6e299f49-hosts-file\") pod \"node-resolver-l6dhh\" (UID: \"e746a856-21ba-4c28-9545-6e6d6e299f49\") " pod="openshift-dns/node-resolver-l6dhh" Jul 06 23:24:50.634787 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634589 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-etc-kubernetes\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.634787 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634647 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs\") pod \"network-metrics-daemon-mdnxn\" (UID: \"f9260140-57c2-4913-adc3-7497f54cd33b\") " pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:24:50.634787 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634664 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-os-release\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.634787 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634688 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-multus-conf-dir\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.634787 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634703 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-multus-daemon-config\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.634787 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634742 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-system-cni-dir\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.634787 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634763 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-host-run-multus-certs\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.634787 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634783 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-cnibin\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.635129 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634805 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-multus-socket-dir-parent\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.635129 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634842 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-host-var-lib-cni-bin\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.635129 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634903 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-multus-cni-dir\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.635129 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.634949 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jdmhg" Jul 06 23:24:50.635129 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.635004 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 23:19:50 +0000 UTC" deadline="2028-04-18 18:10:54.314168362 +0000 UTC" Jul 06 23:24:50.635129 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.635044 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15642h46m3.679128828s" Jul 06 23:24:50.635129 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.635061 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 23:24:50.635584 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.635569 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 23:24:50.635831 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.635807 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 23:24:50.635831 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.635808 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 23:24:50.635969 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.635845 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 23:24:50.635969 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.635813 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 23:24:50.635969 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.635815 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c57r5\"" Jul 06 23:24:50.636127 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.635984 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 23:24:50.636261 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.636244 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.637240 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.637221 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vvbgh\"" Jul 06 23:24:50.637314 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.637264 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 23:24:50.637776 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.637760 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.637864 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.637826 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 23:24:50.638572 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.638558 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 23:24:50.638796 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.638736 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 23:24:50.638949 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.638933 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.639150 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.639134 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 23:24:50.639150 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.639142 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fmgq4\"" Jul 06 23:24:50.639618 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.639599 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:24:50.641760 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.640142 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5crcw\"" Jul 06 23:24:50.641760 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.640170 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 23:24:50.641760 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.640199 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 23:24:50.641760 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.641127 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 23:24:50.641760 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.641192 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 23:24:50.641760 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.641414 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bbhls\"" Jul 06 23:24:50.661591 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.661577 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-85lfm" Jul 06 23:24:50.669485 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.669450 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-85lfm" Jul 06 23:24:50.729379 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.729361 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 23:24:50.735996 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.735976 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e746a856-21ba-4c28-9545-6e6d6e299f49-tmp-dir\") pod \"node-resolver-l6dhh\" (UID: \"e746a856-21ba-4c28-9545-6e6d6e299f49\") " pod="openshift-dns/node-resolver-l6dhh" Jul 06 23:24:50.736059 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736008 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17ddd37f-3cef-44d2-990a-1ca5e4b484fc-host\") pod \"node-ca-kpwsm\" (UID: \"17ddd37f-3cef-44d2-990a-1ca5e4b484fc\") " pod="openshift-image-registry/node-ca-kpwsm" Jul 06 23:24:50.736059 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736028 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17ddd37f-3cef-44d2-990a-1ca5e4b484fc-serviceca\") pod \"node-ca-kpwsm\" (UID: \"17ddd37f-3cef-44d2-990a-1ca5e4b484fc\") " pod="openshift-image-registry/node-ca-kpwsm" Jul 06 23:24:50.736059 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736046 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/23472647-b51d-4d6b-9662-3f56bade3b4d-ovnkube-script-lib\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.736158 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736063 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a181075c-e62d-4224-9cd5-582fb5e04e05-socket-dir\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.736158 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736104 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-sys\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.736220 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736160 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f1332668-fa67-4d07-9a0a-4975eadc0623-os-release\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.736220 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736192 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5cpd\" (UniqueName: \"kubernetes.io/projected/f1332668-fa67-4d07-9a0a-4975eadc0623-kube-api-access-p5cpd\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.736282 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736235 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7xc5\" (UniqueName: \"kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5\") pod \"network-check-target-v8xhl\" (UID: \"8125ec7d-1a89-4270-af40-9b10a7f44cf5\") " pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:24:50.736282 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736273 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-kubelet\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.736346 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736308 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a181075c-e62d-4224-9cd5-582fb5e04e05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.736346 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736323 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e746a856-21ba-4c28-9545-6e6d6e299f49-tmp-dir\") pod \"node-resolver-l6dhh\" (UID: \"e746a856-21ba-4c28-9545-6e6d6e299f49\") " pod="openshift-dns/node-resolver-l6dhh" Jul 06 23:24:50.736346 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736339 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f1332668-fa67-4d07-9a0a-4975eadc0623-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.736440 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736379 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-host-var-lib-kubelet\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.736440 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736415 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-host-var-lib-kubelet\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.736440 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736419 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret\") pod \"global-pull-secret-syncer-rsnfg\" (UID: \"a58729bb-f5a9-4bce-8cfc-3f98f6dd2401\") " pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:24:50.736550 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736458 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e746a856-21ba-4c28-9545-6e6d6e299f49-hosts-file\") pod \"node-resolver-l6dhh\" (UID: \"e746a856-21ba-4c28-9545-6e6d6e299f49\") " pod="openshift-dns/node-resolver-l6dhh" Jul 06 23:24:50.736550 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736529 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e746a856-21ba-4c28-9545-6e6d6e299f49-hosts-file\") pod \"node-resolver-l6dhh\" (UID: \"e746a856-21ba-4c28-9545-6e6d6e299f49\") " pod="openshift-dns/node-resolver-l6dhh" Jul 06 23:24:50.736550 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:50.736543 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:24:50.736660 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736560 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-systemd-units\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.736660 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736593 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs\") pod \"network-metrics-daemon-mdnxn\" (UID: \"f9260140-57c2-4913-adc3-7497f54cd33b\") " pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:24:50.736660 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:50.736600 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret podName:a58729bb-f5a9-4bce-8cfc-3f98f6dd2401 nodeName:}" failed. No retries permitted until 2026-07-06 23:24:51.23657911 +0000 UTC m=+1.950633665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret") pod "global-pull-secret-syncer-rsnfg" (UID: "a58729bb-f5a9-4bce-8cfc-3f98f6dd2401") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:24:50.736660 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736619 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-log-socket\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.736660 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736634 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/23472647-b51d-4d6b-9662-3f56bade3b4d-env-overrides\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.736660 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736652 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/23472647-b51d-4d6b-9662-3f56bade3b4d-ovn-node-metrics-cert\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.736928 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736677 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/025b4529-f609-451b-acc9-19ff4baa1378-tmp\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.736928 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:50.736702 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:24:50.736928 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:50.736780 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs podName:f9260140-57c2-4913-adc3-7497f54cd33b nodeName:}" failed. No retries permitted until 2026-07-06 23:24:51.236767808 +0000 UTC m=+1.950822357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs") pod "network-metrics-daemon-mdnxn" (UID: "f9260140-57c2-4913-adc3-7497f54cd33b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:24:50.736928 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736703 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1332668-fa67-4d07-9a0a-4975eadc0623-system-cni-dir\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.736928 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736804 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-os-release\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.736928 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736829 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/23472647-b51d-4d6b-9662-3f56bade3b4d-ovnkube-config\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.736928 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736854 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvvd9\" (UniqueName: \"kubernetes.io/projected/23472647-b51d-4d6b-9662-3f56bade3b4d-kube-api-access-lvvd9\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.736928 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736882 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-os-release\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.736928 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736895 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-etc-sysctl-conf\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.737353 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736917 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f1332668-fa67-4d07-9a0a-4975eadc0623-cni-binary-copy\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.737353 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736960 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b688q\" (UniqueName: \"kubernetes.io/projected/17ddd37f-3cef-44d2-990a-1ca5e4b484fc-kube-api-access-b688q\") pod \"node-ca-kpwsm\" (UID: \"17ddd37f-3cef-44d2-990a-1ca5e4b484fc\") " pod="openshift-image-registry/node-ca-kpwsm" Jul 06 23:24:50.737353 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.736976 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-etc-kubernetes\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.737353 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737025 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-run-openvswitch\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.737353 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737070 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-run-ovn\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.737353 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737100 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c92b54e3-3f51-45c9-96a7-92f26052db71-konnectivity-ca\") pod \"konnectivity-agent-jdmhg\" (UID: \"c92b54e3-3f51-45c9-96a7-92f26052db71\") " pod="kube-system/konnectivity-agent-jdmhg" Jul 06 23:24:50.737353 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737131 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkdtp\" (UniqueName: \"kubernetes.io/projected/e746a856-21ba-4c28-9545-6e6d6e299f49-kube-api-access-mkdtp\") pod \"node-resolver-l6dhh\" (UID: \"e746a856-21ba-4c28-9545-6e6d6e299f49\") " pod="openshift-dns/node-resolver-l6dhh" Jul 06 23:24:50.737353 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737159 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-etc-openvswitch\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.737353 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737185 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.737353 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737214 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c92b54e3-3f51-45c9-96a7-92f26052db71-agent-certs\") pod \"konnectivity-agent-jdmhg\" (UID: \"c92b54e3-3f51-45c9-96a7-92f26052db71\") " pod="kube-system/konnectivity-agent-jdmhg" Jul 06 23:24:50.737353 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737300 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f1332668-fa67-4d07-9a0a-4975eadc0623-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.737353 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737329 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-cni-binary-copy\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.737353 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737345 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-hostroot\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.737938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737374 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-etc-sysctl-d\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.737938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737384 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-hostroot\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.737938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737453 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f1332668-fa67-4d07-9a0a-4975eadc0623-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.737938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737476 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-host-run-k8s-cni-cncf-io\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.737938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737509 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-host-run-k8s-cni-cncf-io\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.737938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737539 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-host-run-netns\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.737938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737559 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-host-var-lib-cni-multus\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.737938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737583 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-kubelet-config\") pod \"global-pull-secret-syncer-rsnfg\" (UID: \"a58729bb-f5a9-4bce-8cfc-3f98f6dd2401\") " pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:24:50.737938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737610 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-run-netns\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.737938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737629 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-host-run-netns\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.737938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737637 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-cni-bin\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.737938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737673 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-cni-netd\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.737938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737675 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-host-var-lib-cni-multus\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.737938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737701 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-kubelet-config\") pod \"global-pull-secret-syncer-rsnfg\" (UID: \"a58729bb-f5a9-4bce-8cfc-3f98f6dd2401\") " pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:24:50.737938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737694 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brk84\" (UniqueName: \"kubernetes.io/projected/a181075c-e62d-4224-9cd5-582fb5e04e05-kube-api-access-brk84\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.737938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737754 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jk5j\" (UniqueName: \"kubernetes.io/projected/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-kube-api-access-7jk5j\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.737938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737876 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvspz\" (UniqueName: \"kubernetes.io/projected/f9260140-57c2-4913-adc3-7497f54cd33b-kube-api-access-jvspz\") pod \"network-metrics-daemon-mdnxn\" (UID: \"f9260140-57c2-4913-adc3-7497f54cd33b\") " pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:24:50.738664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.737984 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-dbus\") pod \"global-pull-secret-syncer-rsnfg\" (UID: \"a58729bb-f5a9-4bce-8cfc-3f98f6dd2401\") " pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:24:50.738664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738025 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a181075c-e62d-4224-9cd5-582fb5e04e05-device-dir\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.738664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738051 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-var-lib-kubelet\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.738664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738082 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-host\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.738664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738108 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb7mp\" (UniqueName: \"kubernetes.io/projected/025b4529-f609-451b-acc9-19ff4baa1378-kube-api-access-cb7mp\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.738664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738137 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-etc-kubernetes\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.738664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738185 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-node-log\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.738664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738205 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-etc-kubernetes\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.738664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738212 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b2181e0c-0076-401a-a413-252ab0c6d157-iptables-alerter-script\") pod \"iptables-alerter-fwj6n\" (UID: \"b2181e0c-0076-401a-a413-252ab0c6d157\") " pod="openshift-network-operator/iptables-alerter-fwj6n" Jul 06 23:24:50.738664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738157 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-dbus\") pod \"global-pull-secret-syncer-rsnfg\" (UID: \"a58729bb-f5a9-4bce-8cfc-3f98f6dd2401\") " pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:24:50.738664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738243 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78j2c\" (UniqueName: \"kubernetes.io/projected/b2181e0c-0076-401a-a413-252ab0c6d157-kube-api-access-78j2c\") pod \"iptables-alerter-fwj6n\" (UID: \"b2181e0c-0076-401a-a413-252ab0c6d157\") " pod="openshift-network-operator/iptables-alerter-fwj6n" Jul 06 23:24:50.738664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738269 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a181075c-e62d-4224-9cd5-582fb5e04e05-etc-selinux\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.738664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738293 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-etc-systemd\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.738664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738319 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f1332668-fa67-4d07-9a0a-4975eadc0623-cnibin\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.738664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738383 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-multus-conf-dir\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.738664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738416 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-multus-conf-dir\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.738664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738424 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-multus-daemon-config\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.739288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738454 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a181075c-e62d-4224-9cd5-582fb5e04e05-sys-fs\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.739288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738474 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-lib-modules\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.739288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738490 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/025b4529-f609-451b-acc9-19ff4baa1378-etc-tuned\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.739288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738507 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-system-cni-dir\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.739288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738528 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-host-run-multus-certs\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.739288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738552 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-run-systemd\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.739288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738570 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-cni-binary-copy\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.739288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738581 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-var-lib-openvswitch\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.739288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738587 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-host-run-multus-certs\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.739288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738587 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-system-cni-dir\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.739288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738607 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2181e0c-0076-401a-a413-252ab0c6d157-host-slash\") pod \"iptables-alerter-fwj6n\" (UID: \"b2181e0c-0076-401a-a413-252ab0c6d157\") " pod="openshift-network-operator/iptables-alerter-fwj6n" Jul 06 23:24:50.739288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738635 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a181075c-e62d-4224-9cd5-582fb5e04e05-registration-dir\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.739288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738655 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-run\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.739288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738673 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-cnibin\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.739288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738692 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-multus-socket-dir-parent\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.739288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738757 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-cnibin\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.739288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738773 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-multus-socket-dir-parent\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.739932 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738812 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-host-var-lib-cni-bin\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.739932 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738855 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-etc-modprobe-d\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.739932 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738903 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-multus-cni-dir\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.739932 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738925 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-multus-daemon-config\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.739932 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738943 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-host-var-lib-cni-bin\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.739932 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738932 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-slash\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.739932 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.738988 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.739932 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.739000 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-multus-cni-dir\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.739932 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.739017 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-etc-sysconfig\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.744846 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:50.744826 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:24:50.744846 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:50.744848 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:24:50.745008 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:50.744858 2571 projected.go:196] Error preparing data for projected volume kube-api-access-f7xc5 for pod openshift-network-diagnostics/network-check-target-v8xhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:24:50.745008 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:50.744902 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5 podName:8125ec7d-1a89-4270-af40-9b10a7f44cf5 nodeName:}" failed. No retries permitted until 2026-07-06 23:24:51.244890935 +0000 UTC m=+1.958945484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f7xc5" (UniqueName: "kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5") pod "network-check-target-v8xhl" (UID: "8125ec7d-1a89-4270-af40-9b10a7f44cf5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:24:50.745622 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.745577 2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 23:24:50.748964 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.748944 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jk5j\" (UniqueName: \"kubernetes.io/projected/3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5-kube-api-access-7jk5j\") pod \"multus-bgdhp\" (UID: \"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5\") " pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.749068 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.749049 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkdtp\" (UniqueName: \"kubernetes.io/projected/e746a856-21ba-4c28-9545-6e6d6e299f49-kube-api-access-mkdtp\") pod \"node-resolver-l6dhh\" (UID: \"e746a856-21ba-4c28-9545-6e6d6e299f49\") " pod="openshift-dns/node-resolver-l6dhh" Jul 06 23:24:50.749110 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.749049 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvspz\" (UniqueName: \"kubernetes.io/projected/f9260140-57c2-4913-adc3-7497f54cd33b-kube-api-access-jvspz\") pod \"network-metrics-daemon-mdnxn\" (UID: \"f9260140-57c2-4913-adc3-7497f54cd33b\") " pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:24:50.829627 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:50.829596 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7916f8702abca842d06320c25e14c3c0.slice/crio-ccdcc746e97763de95d4950f81d06117d7f8cc31889b6f85f76074760db1a2fc WatchSource:0}: Error finding container ccdcc746e97763de95d4950f81d06117d7f8cc31889b6f85f76074760db1a2fc: Status 404 returned error can't find the container with id ccdcc746e97763de95d4950f81d06117d7f8cc31889b6f85f76074760db1a2fc Jul 06 23:24:50.830634 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:50.830610 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e3d258c0c669908efda319c624e11f.slice/crio-f150f19278aa085d0f8b197e2838239cc3f771d7386d44b745f71f53e43c9613 WatchSource:0}: Error finding container f150f19278aa085d0f8b197e2838239cc3f771d7386d44b745f71f53e43c9613: Status 404 returned error can't find the container with id f150f19278aa085d0f8b197e2838239cc3f771d7386d44b745f71f53e43c9613 Jul 06 23:24:50.837229 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.837212 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 23:24:50.840037 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840014 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78j2c\" (UniqueName: \"kubernetes.io/projected/b2181e0c-0076-401a-a413-252ab0c6d157-kube-api-access-78j2c\") pod \"iptables-alerter-fwj6n\" (UID: \"b2181e0c-0076-401a-a413-252ab0c6d157\") " pod="openshift-network-operator/iptables-alerter-fwj6n" Jul 06 23:24:50.840138 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840045 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a181075c-e62d-4224-9cd5-582fb5e04e05-etc-selinux\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.840138 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840063 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-etc-systemd\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.840138 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840089 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f1332668-fa67-4d07-9a0a-4975eadc0623-cnibin\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.840298 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840161 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f1332668-fa67-4d07-9a0a-4975eadc0623-cnibin\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.840298 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840162 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-etc-systemd\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.840298 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840171 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a181075c-e62d-4224-9cd5-582fb5e04e05-etc-selinux\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.840298 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840218 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a181075c-e62d-4224-9cd5-582fb5e04e05-sys-fs\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.840298 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840247 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-lib-modules\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.840298 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840274 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a181075c-e62d-4224-9cd5-582fb5e04e05-sys-fs\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.840298 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840275 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/025b4529-f609-451b-acc9-19ff4baa1378-etc-tuned\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.840664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840321 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-run-systemd\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.840664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840342 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-var-lib-openvswitch\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.840664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840357 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2181e0c-0076-401a-a413-252ab0c6d157-host-slash\") pod \"iptables-alerter-fwj6n\" (UID: \"b2181e0c-0076-401a-a413-252ab0c6d157\") " pod="openshift-network-operator/iptables-alerter-fwj6n" Jul 06 23:24:50.840664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840381 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a181075c-e62d-4224-9cd5-582fb5e04e05-registration-dir\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.840664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840389 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-var-lib-openvswitch\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.840664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840394 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-run-systemd\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.840664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840408 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2181e0c-0076-401a-a413-252ab0c6d157-host-slash\") pod \"iptables-alerter-fwj6n\" (UID: \"b2181e0c-0076-401a-a413-252ab0c6d157\") " pod="openshift-network-operator/iptables-alerter-fwj6n" Jul 06 23:24:50.840664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840356 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-lib-modules\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.840664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840448 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a181075c-e62d-4224-9cd5-582fb5e04e05-registration-dir\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.840664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840476 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-run\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.840664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840495 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-etc-modprobe-d\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.840664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840511 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-slash\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.840664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840526 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.840664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840542 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-etc-sysconfig\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.840664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840557 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17ddd37f-3cef-44d2-990a-1ca5e4b484fc-host\") pod \"node-ca-kpwsm\" (UID: \"17ddd37f-3cef-44d2-990a-1ca5e4b484fc\") " pod="openshift-image-registry/node-ca-kpwsm" Jul 06 23:24:50.840664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840575 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-slash\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.840664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840588 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.841517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840602 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17ddd37f-3cef-44d2-990a-1ca5e4b484fc-host\") pod \"node-ca-kpwsm\" (UID: \"17ddd37f-3cef-44d2-990a-1ca5e4b484fc\") " pod="openshift-image-registry/node-ca-kpwsm" Jul 06 23:24:50.841517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840619 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-run\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.841517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840623 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17ddd37f-3cef-44d2-990a-1ca5e4b484fc-serviceca\") pod \"node-ca-kpwsm\" (UID: \"17ddd37f-3cef-44d2-990a-1ca5e4b484fc\") " pod="openshift-image-registry/node-ca-kpwsm" Jul 06 23:24:50.841517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840630 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-etc-sysconfig\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.841517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840649 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/23472647-b51d-4d6b-9662-3f56bade3b4d-ovnkube-script-lib\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.841517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840619 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-etc-modprobe-d\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.841517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840667 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a181075c-e62d-4224-9cd5-582fb5e04e05-socket-dir\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.841517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840686 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-sys\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.841517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840722 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f1332668-fa67-4d07-9a0a-4975eadc0623-os-release\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.841517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840748 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5cpd\" (UniqueName: \"kubernetes.io/projected/f1332668-fa67-4d07-9a0a-4975eadc0623-kube-api-access-p5cpd\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.841517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840813 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f1332668-fa67-4d07-9a0a-4975eadc0623-os-release\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.841517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840820 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a181075c-e62d-4224-9cd5-582fb5e04e05-socket-dir\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.841517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840834 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-sys\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.841517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840877 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-kubelet\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.841517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840897 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-kubelet\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.841517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.840906 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a181075c-e62d-4224-9cd5-582fb5e04e05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.841517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841002 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a181075c-e62d-4224-9cd5-582fb5e04e05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.841517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841104 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17ddd37f-3cef-44d2-990a-1ca5e4b484fc-serviceca\") pod \"node-ca-kpwsm\" (UID: \"17ddd37f-3cef-44d2-990a-1ca5e4b484fc\") " pod="openshift-image-registry/node-ca-kpwsm" Jul 06 23:24:50.842575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841144 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f1332668-fa67-4d07-9a0a-4975eadc0623-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.842575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841194 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-systemd-units\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.842575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841217 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/23472647-b51d-4d6b-9662-3f56bade3b4d-ovnkube-script-lib\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.842575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841233 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-log-socket\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.842575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841263 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-systemd-units\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.842575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841268 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/23472647-b51d-4d6b-9662-3f56bade3b4d-env-overrides\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.842575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841298 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-log-socket\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.842575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841297 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/23472647-b51d-4d6b-9662-3f56bade3b4d-ovn-node-metrics-cert\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.842575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841328 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/025b4529-f609-451b-acc9-19ff4baa1378-tmp\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.842575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841349 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1332668-fa67-4d07-9a0a-4975eadc0623-system-cni-dir\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.842575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841380 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/23472647-b51d-4d6b-9662-3f56bade3b4d-ovnkube-config\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.842575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841404 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvvd9\" (UniqueName: \"kubernetes.io/projected/23472647-b51d-4d6b-9662-3f56bade3b4d-kube-api-access-lvvd9\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.842575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841429 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-etc-sysctl-conf\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.842575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841455 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f1332668-fa67-4d07-9a0a-4975eadc0623-cni-binary-copy\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.842575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841480 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b688q\" (UniqueName: \"kubernetes.io/projected/17ddd37f-3cef-44d2-990a-1ca5e4b484fc-kube-api-access-b688q\") pod \"node-ca-kpwsm\" (UID: \"17ddd37f-3cef-44d2-990a-1ca5e4b484fc\") " pod="openshift-image-registry/node-ca-kpwsm" Jul 06 23:24:50.842575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841507 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-etc-kubernetes\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.842575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841535 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-run-openvswitch\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.843532 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841562 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-run-ovn\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.843532 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841565 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f1332668-fa67-4d07-9a0a-4975eadc0623-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.843532 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841587 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c92b54e3-3f51-45c9-96a7-92f26052db71-konnectivity-ca\") pod \"konnectivity-agent-jdmhg\" (UID: \"c92b54e3-3f51-45c9-96a7-92f26052db71\") " pod="kube-system/konnectivity-agent-jdmhg" Jul 06 23:24:50.843532 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841615 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-etc-openvswitch\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.843532 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841640 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.843532 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841660 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c92b54e3-3f51-45c9-96a7-92f26052db71-agent-certs\") pod \"konnectivity-agent-jdmhg\" (UID: \"c92b54e3-3f51-45c9-96a7-92f26052db71\") " pod="kube-system/konnectivity-agent-jdmhg" Jul 06 23:24:50.843532 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841679 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f1332668-fa67-4d07-9a0a-4975eadc0623-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.843532 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841714 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-etc-sysctl-d\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.843532 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841739 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f1332668-fa67-4d07-9a0a-4975eadc0623-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.843532 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841781 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-run-ovn\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.843532 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841861 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-etc-openvswitch\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.843532 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841875 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/23472647-b51d-4d6b-9662-3f56bade3b4d-env-overrides\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.843532 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841931 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.843532 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.842038 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f1332668-fa67-4d07-9a0a-4975eadc0623-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.843532 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.842110 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f1332668-fa67-4d07-9a0a-4975eadc0623-cni-binary-copy\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.843532 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.842183 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1332668-fa67-4d07-9a0a-4975eadc0623-system-cni-dir\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.843532 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.842218 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-etc-sysctl-d\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.844348 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.842668 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/23472647-b51d-4d6b-9662-3f56bade3b4d-ovnkube-config\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.844348 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.842784 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f1332668-fa67-4d07-9a0a-4975eadc0623-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.844348 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843125 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/025b4529-f609-451b-acc9-19ff4baa1378-etc-tuned\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.844348 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843128 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-run-openvswitch\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.844348 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.841792 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-run-netns\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.844348 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843182 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-etc-kubernetes\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.844348 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843201 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-cni-bin\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.844348 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843204 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-etc-sysctl-conf\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.844348 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843254 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-cni-bin\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.844348 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843256 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-run-netns\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.844348 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843284 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-cni-netd\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.844348 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843319 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brk84\" (UniqueName: \"kubernetes.io/projected/a181075c-e62d-4224-9cd5-582fb5e04e05-kube-api-access-brk84\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.844348 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843352 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a181075c-e62d-4224-9cd5-582fb5e04e05-device-dir\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.844348 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843402 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-host-cni-netd\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.844348 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843426 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-var-lib-kubelet\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.844348 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843454 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a181075c-e62d-4224-9cd5-582fb5e04e05-device-dir\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.844348 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843458 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-host\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.844987 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843492 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb7mp\" (UniqueName: \"kubernetes.io/projected/025b4529-f609-451b-acc9-19ff4baa1378-kube-api-access-cb7mp\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.844987 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843525 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-node-log\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.844987 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843508 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-host\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.844987 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843550 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b2181e0c-0076-401a-a413-252ab0c6d157-iptables-alerter-script\") pod \"iptables-alerter-fwj6n\" (UID: \"b2181e0c-0076-401a-a413-252ab0c6d157\") " pod="openshift-network-operator/iptables-alerter-fwj6n" Jul 06 23:24:50.844987 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843587 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/025b4529-f609-451b-acc9-19ff4baa1378-var-lib-kubelet\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.844987 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843631 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/23472647-b51d-4d6b-9662-3f56bade3b4d-node-log\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.844987 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.843897 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c92b54e3-3f51-45c9-96a7-92f26052db71-konnectivity-ca\") pod \"konnectivity-agent-jdmhg\" (UID: \"c92b54e3-3f51-45c9-96a7-92f26052db71\") " pod="kube-system/konnectivity-agent-jdmhg" Jul 06 23:24:50.844987 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.844111 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/23472647-b51d-4d6b-9662-3f56bade3b4d-ovn-node-metrics-cert\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.844987 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.844126 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b2181e0c-0076-401a-a413-252ab0c6d157-iptables-alerter-script\") pod \"iptables-alerter-fwj6n\" (UID: \"b2181e0c-0076-401a-a413-252ab0c6d157\") " pod="openshift-network-operator/iptables-alerter-fwj6n" Jul 06 23:24:50.844987 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.844852 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/025b4529-f609-451b-acc9-19ff4baa1378-tmp\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.844987 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.844911 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c92b54e3-3f51-45c9-96a7-92f26052db71-agent-certs\") pod \"konnectivity-agent-jdmhg\" (UID: \"c92b54e3-3f51-45c9-96a7-92f26052db71\") " pod="kube-system/konnectivity-agent-jdmhg" Jul 06 23:24:50.852392 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.852368 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb7mp\" (UniqueName: \"kubernetes.io/projected/025b4529-f609-451b-acc9-19ff4baa1378-kube-api-access-cb7mp\") pod \"tuned-4bk8b\" (UID: \"025b4529-f609-451b-acc9-19ff4baa1378\") " pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:50.852853 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.852830 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5cpd\" (UniqueName: \"kubernetes.io/projected/f1332668-fa67-4d07-9a0a-4975eadc0623-kube-api-access-p5cpd\") pod \"multus-additional-cni-plugins-bvl5t\" (UID: \"f1332668-fa67-4d07-9a0a-4975eadc0623\") " pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:50.852971 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.852850 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brk84\" (UniqueName: \"kubernetes.io/projected/a181075c-e62d-4224-9cd5-582fb5e04e05-kube-api-access-brk84\") pod \"aws-ebs-csi-driver-node-2pht6\" (UID: \"a181075c-e62d-4224-9cd5-582fb5e04e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:50.852971 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.852884 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b688q\" (UniqueName: \"kubernetes.io/projected/17ddd37f-3cef-44d2-990a-1ca5e4b484fc-kube-api-access-b688q\") pod \"node-ca-kpwsm\" (UID: \"17ddd37f-3cef-44d2-990a-1ca5e4b484fc\") " pod="openshift-image-registry/node-ca-kpwsm" Jul 06 23:24:50.853092 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.853028 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78j2c\" (UniqueName: \"kubernetes.io/projected/b2181e0c-0076-401a-a413-252ab0c6d157-kube-api-access-78j2c\") pod \"iptables-alerter-fwj6n\" (UID: \"b2181e0c-0076-401a-a413-252ab0c6d157\") " pod="openshift-network-operator/iptables-alerter-fwj6n" Jul 06 23:24:50.853198 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.853179 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvvd9\" (UniqueName: \"kubernetes.io/projected/23472647-b51d-4d6b-9662-3f56bade3b4d-kube-api-access-lvvd9\") pod \"ovnkube-node-c7gg2\" (UID: \"23472647-b51d-4d6b-9662-3f56bade3b4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:50.950739 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.950628 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bgdhp" Jul 06 23:24:50.960786 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.960751 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l6dhh" Jul 06 23:24:50.961119 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:50.961095 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4a0f6b_bcbe_4399_b6b4_f31c9f96b1f5.slice/crio-589ad04800aaf8a954219c24372ed0ca33f13a444f9a31e7ad1f12626d3744c2 WatchSource:0}: Error finding container 589ad04800aaf8a954219c24372ed0ca33f13a444f9a31e7ad1f12626d3744c2: Status 404 returned error can't find the container with id 589ad04800aaf8a954219c24372ed0ca33f13a444f9a31e7ad1f12626d3744c2 Jul 06 23:24:50.969612 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:50.969586 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode746a856_21ba_4c28_9545_6e6d6e299f49.slice/crio-e1b4b74fefc054e041ad5f27847a88b78bba6803e90863c1e3bfcf248b00de98 WatchSource:0}: Error finding container e1b4b74fefc054e041ad5f27847a88b78bba6803e90863c1e3bfcf248b00de98: Status 404 returned error can't find the container with id e1b4b74fefc054e041ad5f27847a88b78bba6803e90863c1e3bfcf248b00de98 Jul 06 23:24:50.984665 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.984646 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kpwsm" Jul 06 23:24:50.993839 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:50.993817 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ddd37f_3cef_44d2_990a_1ca5e4b484fc.slice/crio-4d0b40bc1bde5bb835c346b472625d500f4761e4a3feff589f1e51c2e0543d50 WatchSource:0}: Error finding container 4d0b40bc1bde5bb835c346b472625d500f4761e4a3feff589f1e51c2e0543d50: Status 404 returned error can't find the container with id 4d0b40bc1bde5bb835c346b472625d500f4761e4a3feff589f1e51c2e0543d50 Jul 06 23:24:50.995532 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:50.995512 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fwj6n" Jul 06 23:24:51.002159 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.002141 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:24:51.003830 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:51.003811 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2181e0c_0076_401a_a413_252ab0c6d157.slice/crio-370a4c2b7600c3d190d4259e9dfc33eb032b903fb9b57d94d6d3251d54e63178 WatchSource:0}: Error finding container 370a4c2b7600c3d190d4259e9dfc33eb032b903fb9b57d94d6d3251d54e63178: Status 404 returned error can't find the container with id 370a4c2b7600c3d190d4259e9dfc33eb032b903fb9b57d94d6d3251d54e63178 Jul 06 23:24:51.011632 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:51.011612 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23472647_b51d_4d6b_9662_3f56bade3b4d.slice/crio-956b22489d14b97dc163796de80597cc23c84760c2cf97de50d69611c48a11dd WatchSource:0}: Error finding container 956b22489d14b97dc163796de80597cc23c84760c2cf97de50d69611c48a11dd: Status 404 returned error can't find the container with id 956b22489d14b97dc163796de80597cc23c84760c2cf97de50d69611c48a11dd Jul 06 23:24:51.015601 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.015583 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jdmhg" Jul 06 23:24:51.024144 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:51.024122 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92b54e3_3f51_45c9_96a7_92f26052db71.slice/crio-667ec3399dfa97f530fc50b88e77e662f488c5b1fd726f1d13533a1acde8ba77 WatchSource:0}: Error finding container 667ec3399dfa97f530fc50b88e77e662f488c5b1fd726f1d13533a1acde8ba77: Status 404 returned error can't find the container with id 667ec3399dfa97f530fc50b88e77e662f488c5b1fd726f1d13533a1acde8ba77 Jul 06 23:24:51.040903 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.040881 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" Jul 06 23:24:51.050034 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:51.050004 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda181075c_e62d_4224_9cd5_582fb5e04e05.slice/crio-b8cf9a407905e84b19cdcfe42bbf8679a938ff59858af944b177dd0a76ee0542 WatchSource:0}: Error finding container b8cf9a407905e84b19cdcfe42bbf8679a938ff59858af944b177dd0a76ee0542: Status 404 returned error can't find the container with id b8cf9a407905e84b19cdcfe42bbf8679a938ff59858af944b177dd0a76ee0542 Jul 06 23:24:51.062367 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.062348 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" Jul 06 23:24:51.067326 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.067306 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bvl5t" Jul 06 23:24:51.071144 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:51.071124 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025b4529_f609_451b_acc9_19ff4baa1378.slice/crio-985fc44805b304ca2793bf022e98d452316497e38c9ae10fa944e7e311b76f0f WatchSource:0}: Error finding container 985fc44805b304ca2793bf022e98d452316497e38c9ae10fa944e7e311b76f0f: Status 404 returned error can't find the container with id 985fc44805b304ca2793bf022e98d452316497e38c9ae10fa944e7e311b76f0f Jul 06 23:24:51.076101 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:24:51.076059 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1332668_fa67_4d07_9a0a_4975eadc0623.slice/crio-38b7b64fdd83d09fd0eeab757dfd0eebeb8d3feddbf2bd9f1f864a69da26646d WatchSource:0}: Error finding container 38b7b64fdd83d09fd0eeab757dfd0eebeb8d3feddbf2bd9f1f864a69da26646d: Status 404 returned error can't find the container with id 38b7b64fdd83d09fd0eeab757dfd0eebeb8d3feddbf2bd9f1f864a69da26646d Jul 06 23:24:51.198260 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.198230 2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:24:51.246476 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.246379 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7xc5\" (UniqueName: \"kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5\") pod \"network-check-target-v8xhl\" (UID: \"8125ec7d-1a89-4270-af40-9b10a7f44cf5\") " pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:24:51.246476 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.246432 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret\") pod \"global-pull-secret-syncer-rsnfg\" (UID: \"a58729bb-f5a9-4bce-8cfc-3f98f6dd2401\") " pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:24:51.246476 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.246464 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs\") pod \"network-metrics-daemon-mdnxn\" (UID: \"f9260140-57c2-4913-adc3-7497f54cd33b\") " pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:24:51.246758 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:51.246567 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:24:51.246758 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:51.246589 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:24:51.246758 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:51.246596 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:24:51.246758 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:51.246604 2571 projected.go:196] Error preparing data for projected volume kube-api-access-f7xc5 for pod openshift-network-diagnostics/network-check-target-v8xhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:24:51.246758 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:51.246607 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:24:51.246758 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:51.246663 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs podName:f9260140-57c2-4913-adc3-7497f54cd33b nodeName:}" failed. No retries permitted until 2026-07-06 23:24:52.246643296 +0000 UTC m=+2.960697852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs") pod "network-metrics-daemon-mdnxn" (UID: "f9260140-57c2-4913-adc3-7497f54cd33b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:24:51.246758 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:51.246683 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5 podName:8125ec7d-1a89-4270-af40-9b10a7f44cf5 nodeName:}" failed. No retries permitted until 2026-07-06 23:24:52.246671155 +0000 UTC m=+2.960725708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7xc5" (UniqueName: "kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5") pod "network-check-target-v8xhl" (UID: "8125ec7d-1a89-4270-af40-9b10a7f44cf5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:24:51.246758 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:51.246699 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret podName:a58729bb-f5a9-4bce-8cfc-3f98f6dd2401 nodeName:}" failed. No retries permitted until 2026-07-06 23:24:52.246689354 +0000 UTC m=+2.960743905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret") pod "global-pull-secret-syncer-rsnfg" (UID: "a58729bb-f5a9-4bce-8cfc-3f98f6dd2401") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:24:51.675886 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.672757 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 23:19:51 +0000 UTC" deadline="2028-02-25 08:31:26.117867323 +0000 UTC" Jul 06 23:24:51.675886 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.672816 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14361h6m34.445055653s" Jul 06 23:24:51.780159 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.780124 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:24:51.780353 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:51.780288 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:24:51.812901 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.812838 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" event={"ID":"025b4529-f609-451b-acc9-19ff4baa1378","Type":"ContainerStarted","Data":"985fc44805b304ca2793bf022e98d452316497e38c9ae10fa944e7e311b76f0f"} Jul 06 23:24:51.847850 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.847750 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jdmhg" event={"ID":"c92b54e3-3f51-45c9-96a7-92f26052db71","Type":"ContainerStarted","Data":"667ec3399dfa97f530fc50b88e77e662f488c5b1fd726f1d13533a1acde8ba77"} Jul 06 23:24:51.858753 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.858719 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" event={"ID":"23472647-b51d-4d6b-9662-3f56bade3b4d","Type":"ContainerStarted","Data":"956b22489d14b97dc163796de80597cc23c84760c2cf97de50d69611c48a11dd"} Jul 06 23:24:51.877491 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.875676 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kpwsm" event={"ID":"17ddd37f-3cef-44d2-990a-1ca5e4b484fc","Type":"ContainerStarted","Data":"4d0b40bc1bde5bb835c346b472625d500f4761e4a3feff589f1e51c2e0543d50"} Jul 06 23:24:51.880725 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.880493 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" event={"ID":"a181075c-e62d-4224-9cd5-582fb5e04e05","Type":"ContainerStarted","Data":"b8cf9a407905e84b19cdcfe42bbf8679a938ff59858af944b177dd0a76ee0542"} Jul 06 23:24:51.901613 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.901441 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fwj6n" event={"ID":"b2181e0c-0076-401a-a413-252ab0c6d157","Type":"ContainerStarted","Data":"370a4c2b7600c3d190d4259e9dfc33eb032b903fb9b57d94d6d3251d54e63178"} Jul 06 23:24:51.916296 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.916262 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l6dhh" event={"ID":"e746a856-21ba-4c28-9545-6e6d6e299f49","Type":"ContainerStarted","Data":"e1b4b74fefc054e041ad5f27847a88b78bba6803e90863c1e3bfcf248b00de98"} Jul 06 23:24:51.927920 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.927839 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bgdhp" event={"ID":"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5","Type":"ContainerStarted","Data":"589ad04800aaf8a954219c24372ed0ca33f13a444f9a31e7ad1f12626d3744c2"} Jul 06 23:24:51.940367 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.940342 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:24:51.949291 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.949258 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal" event={"ID":"84e3d258c0c669908efda319c624e11f","Type":"ContainerStarted","Data":"f150f19278aa085d0f8b197e2838239cc3f771d7386d44b745f71f53e43c9613"} Jul 06 23:24:51.956659 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.956544 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal" event={"ID":"7916f8702abca842d06320c25e14c3c0","Type":"ContainerStarted","Data":"ccdcc746e97763de95d4950f81d06117d7f8cc31889b6f85f76074760db1a2fc"} Jul 06 23:24:51.971878 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:51.971841 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvl5t" event={"ID":"f1332668-fa67-4d07-9a0a-4975eadc0623","Type":"ContainerStarted","Data":"38b7b64fdd83d09fd0eeab757dfd0eebeb8d3feddbf2bd9f1f864a69da26646d"} Jul 06 23:24:52.253515 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:52.253413 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7xc5\" (UniqueName: \"kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5\") pod \"network-check-target-v8xhl\" (UID: \"8125ec7d-1a89-4270-af40-9b10a7f44cf5\") " pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:24:52.253515 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:52.253476 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret\") pod \"global-pull-secret-syncer-rsnfg\" (UID: \"a58729bb-f5a9-4bce-8cfc-3f98f6dd2401\") " pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:24:52.253515 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:52.253511 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs\") pod \"network-metrics-daemon-mdnxn\" (UID: \"f9260140-57c2-4913-adc3-7497f54cd33b\") " pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:24:52.253942 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:52.253685 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:24:52.253942 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:52.253755 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs podName:f9260140-57c2-4913-adc3-7497f54cd33b nodeName:}" failed. No retries permitted until 2026-07-06 23:24:54.25373268 +0000 UTC m=+4.967787239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs") pod "network-metrics-daemon-mdnxn" (UID: "f9260140-57c2-4913-adc3-7497f54cd33b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:24:52.254303 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:52.254281 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:24:52.254394 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:52.254310 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:24:52.254394 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:52.254325 2571 projected.go:196] Error preparing data for projected volume kube-api-access-f7xc5 for pod openshift-network-diagnostics/network-check-target-v8xhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:24:52.254394 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:52.254378 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5 podName:8125ec7d-1a89-4270-af40-9b10a7f44cf5 nodeName:}" failed. No retries permitted until 2026-07-06 23:24:54.254360262 +0000 UTC m=+4.968414817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7xc5" (UniqueName: "kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5") pod "network-check-target-v8xhl" (UID: "8125ec7d-1a89-4270-af40-9b10a7f44cf5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:24:52.254571 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:52.254456 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:24:52.254571 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:52.254493 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret podName:a58729bb-f5a9-4bce-8cfc-3f98f6dd2401 nodeName:}" failed. No retries permitted until 2026-07-06 23:24:54.254479839 +0000 UTC m=+4.968534399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret") pod "global-pull-secret-syncer-rsnfg" (UID: "a58729bb-f5a9-4bce-8cfc-3f98f6dd2401") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:24:52.673924 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:52.673874 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 23:19:51 +0000 UTC" deadline="2028-02-16 06:25:20.022536891 +0000 UTC" Jul 06 23:24:52.673924 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:52.673915 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14143h0m27.348627205s" Jul 06 23:24:52.780728 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:52.780693 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:24:52.781148 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:52.780873 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsnfg" podUID="a58729bb-f5a9-4bce-8cfc-3f98f6dd2401" Jul 06 23:24:52.781290 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:52.781258 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:24:52.782169 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:52.781372 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdnxn" podUID="f9260140-57c2-4913-adc3-7497f54cd33b" Jul 06 23:24:53.171511 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:53.171482 2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:24:53.782755 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:53.782717 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:24:53.783266 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:53.782874 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:24:54.279185 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:54.278452 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7xc5\" (UniqueName: \"kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5\") pod \"network-check-target-v8xhl\" (UID: \"8125ec7d-1a89-4270-af40-9b10a7f44cf5\") " pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:24:54.279185 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:54.278514 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret\") pod \"global-pull-secret-syncer-rsnfg\" (UID: \"a58729bb-f5a9-4bce-8cfc-3f98f6dd2401\") " pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:24:54.279185 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:54.278551 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs\") pod \"network-metrics-daemon-mdnxn\" (UID: \"f9260140-57c2-4913-adc3-7497f54cd33b\") " pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:24:54.279185 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:54.278645 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:24:54.279185 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:54.278648 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:24:54.279185 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:54.278675 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:24:54.279185 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:54.278691 2571 projected.go:196] Error preparing data for projected volume kube-api-access-f7xc5 for pod openshift-network-diagnostics/network-check-target-v8xhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:24:54.279185 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:54.278716 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs podName:f9260140-57c2-4913-adc3-7497f54cd33b nodeName:}" failed. No retries permitted until 2026-07-06 23:24:58.278696863 +0000 UTC m=+8.992751413 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs") pod "network-metrics-daemon-mdnxn" (UID: "f9260140-57c2-4913-adc3-7497f54cd33b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:24:54.279185 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:54.278763 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5 podName:8125ec7d-1a89-4270-af40-9b10a7f44cf5 nodeName:}" failed. No retries permitted until 2026-07-06 23:24:58.278750704 +0000 UTC m=+8.992805274 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7xc5" (UniqueName: "kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5") pod "network-check-target-v8xhl" (UID: "8125ec7d-1a89-4270-af40-9b10a7f44cf5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:24:54.279185 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:54.278781 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:24:54.279185 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:54.278875 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret podName:a58729bb-f5a9-4bce-8cfc-3f98f6dd2401 nodeName:}" failed. No retries permitted until 2026-07-06 23:24:58.278854902 +0000 UTC m=+8.992909465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret") pod "global-pull-secret-syncer-rsnfg" (UID: "a58729bb-f5a9-4bce-8cfc-3f98f6dd2401") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:24:54.780621 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:54.780581 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:24:54.780841 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:54.780736 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsnfg" podUID="a58729bb-f5a9-4bce-8cfc-3f98f6dd2401" Jul 06 23:24:54.781269 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:54.781167 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:24:54.781371 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:54.781281 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdnxn" podUID="f9260140-57c2-4913-adc3-7497f54cd33b" Jul 06 23:24:55.782694 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:55.782643 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:24:55.783174 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:55.782765 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:24:56.780490 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:56.780455 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:24:56.780678 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:56.780510 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:24:56.780678 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:56.780602 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsnfg" podUID="a58729bb-f5a9-4bce-8cfc-3f98f6dd2401" Jul 06 23:24:56.780808 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:56.780784 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdnxn" podUID="f9260140-57c2-4913-adc3-7497f54cd33b" Jul 06 23:24:57.780387 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:57.779891 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:24:57.780387 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:57.780003 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:24:58.316925 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:58.316880 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7xc5\" (UniqueName: \"kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5\") pod \"network-check-target-v8xhl\" (UID: \"8125ec7d-1a89-4270-af40-9b10a7f44cf5\") " pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:24:58.317129 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:58.316943 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret\") pod \"global-pull-secret-syncer-rsnfg\" (UID: \"a58729bb-f5a9-4bce-8cfc-3f98f6dd2401\") " pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:24:58.317129 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:58.316977 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs\") pod \"network-metrics-daemon-mdnxn\" (UID: \"f9260140-57c2-4913-adc3-7497f54cd33b\") " pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:24:58.317264 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:58.317145 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:24:58.317264 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:58.317203 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs podName:f9260140-57c2-4913-adc3-7497f54cd33b nodeName:}" failed. No retries permitted until 2026-07-06 23:25:06.317188999 +0000 UTC m=+17.031243546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs") pod "network-metrics-daemon-mdnxn" (UID: "f9260140-57c2-4913-adc3-7497f54cd33b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:24:58.317629 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:58.317612 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:24:58.317748 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:58.317637 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:24:58.317748 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:58.317649 2571 projected.go:196] Error preparing data for projected volume kube-api-access-f7xc5 for pod openshift-network-diagnostics/network-check-target-v8xhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:24:58.317748 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:58.317695 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5 podName:8125ec7d-1a89-4270-af40-9b10a7f44cf5 nodeName:}" failed. No retries permitted until 2026-07-06 23:25:06.317679949 +0000 UTC m=+17.031734499 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7xc5" (UniqueName: "kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5") pod "network-check-target-v8xhl" (UID: "8125ec7d-1a89-4270-af40-9b10a7f44cf5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:24:58.317962 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:58.317788 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:24:58.317962 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:58.317824 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret podName:a58729bb-f5a9-4bce-8cfc-3f98f6dd2401 nodeName:}" failed. No retries permitted until 2026-07-06 23:25:06.317812937 +0000 UTC m=+17.031867488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret") pod "global-pull-secret-syncer-rsnfg" (UID: "a58729bb-f5a9-4bce-8cfc-3f98f6dd2401") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:24:58.780125 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:58.780092 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:24:58.780291 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:58.780266 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsnfg" podUID="a58729bb-f5a9-4bce-8cfc-3f98f6dd2401" Jul 06 23:24:58.780695 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:58.780677 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:24:58.781091 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:58.780787 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdnxn" podUID="f9260140-57c2-4913-adc3-7497f54cd33b" Jul 06 23:24:59.780411 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:24:59.780380 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:24:59.780552 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:24:59.780457 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:25:00.779775 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:00.779748 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:00.780205 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:00.779789 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:00.780205 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:00.779871 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsnfg" podUID="a58729bb-f5a9-4bce-8cfc-3f98f6dd2401" Jul 06 23:25:00.780205 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:00.780007 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdnxn" podUID="f9260140-57c2-4913-adc3-7497f54cd33b" Jul 06 23:25:01.780820 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:01.780781 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:01.781213 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:01.780896 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:25:02.780357 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:02.780325 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:02.780537 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:02.780331 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:02.780537 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:02.780461 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsnfg" podUID="a58729bb-f5a9-4bce-8cfc-3f98f6dd2401" Jul 06 23:25:02.780630 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:02.780532 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdnxn" podUID="f9260140-57c2-4913-adc3-7497f54cd33b" Jul 06 23:25:03.780869 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:03.780643 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:03.781315 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:03.780980 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:25:04.779797 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:04.779761 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:04.779797 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:04.779798 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:04.780034 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:04.779871 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsnfg" podUID="a58729bb-f5a9-4bce-8cfc-3f98f6dd2401" Jul 06 23:25:04.780034 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:04.780012 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdnxn" podUID="f9260140-57c2-4913-adc3-7497f54cd33b" Jul 06 23:25:05.780730 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:05.780683 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:05.781171 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:05.780830 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:25:06.378922 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:06.378887 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7xc5\" (UniqueName: \"kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5\") pod \"network-check-target-v8xhl\" (UID: \"8125ec7d-1a89-4270-af40-9b10a7f44cf5\") " pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:06.378922 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:06.378926 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret\") pod \"global-pull-secret-syncer-rsnfg\" (UID: \"a58729bb-f5a9-4bce-8cfc-3f98f6dd2401\") " pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:06.379195 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:06.378951 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs\") pod \"network-metrics-daemon-mdnxn\" (UID: \"f9260140-57c2-4913-adc3-7497f54cd33b\") " pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:06.379195 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:06.379042 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:25:06.379195 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:06.379056 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:25:06.379195 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:06.379061 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:25:06.379195 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:06.379073 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:25:06.379195 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:06.379089 2571 projected.go:196] Error preparing data for projected volume kube-api-access-f7xc5 for pod openshift-network-diagnostics/network-check-target-v8xhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:25:06.379195 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:06.379101 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs podName:f9260140-57c2-4913-adc3-7497f54cd33b nodeName:}" failed. No retries permitted until 2026-07-06 23:25:22.379083662 +0000 UTC m=+33.093138211 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs") pod "network-metrics-daemon-mdnxn" (UID: "f9260140-57c2-4913-adc3-7497f54cd33b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:25:06.379195 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:06.379124 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5 podName:8125ec7d-1a89-4270-af40-9b10a7f44cf5 nodeName:}" failed. No retries permitted until 2026-07-06 23:25:22.379115338 +0000 UTC m=+33.093169885 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7xc5" (UniqueName: "kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5") pod "network-check-target-v8xhl" (UID: "8125ec7d-1a89-4270-af40-9b10a7f44cf5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:25:06.379195 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:06.379155 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret podName:a58729bb-f5a9-4bce-8cfc-3f98f6dd2401 nodeName:}" failed. No retries permitted until 2026-07-06 23:25:22.379137922 +0000 UTC m=+33.093192474 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret") pod "global-pull-secret-syncer-rsnfg" (UID: "a58729bb-f5a9-4bce-8cfc-3f98f6dd2401") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:25:06.780261 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:06.780228 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:06.780261 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:06.780261 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:06.780562 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:06.780382 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsnfg" podUID="a58729bb-f5a9-4bce-8cfc-3f98f6dd2401" Jul 06 23:25:06.780562 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:06.780485 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdnxn" podUID="f9260140-57c2-4913-adc3-7497f54cd33b" Jul 06 23:25:07.784507 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:07.780874 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:07.784507 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:07.781039 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:25:08.780299 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:08.780268 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:08.780506 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:08.780429 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdnxn" podUID="f9260140-57c2-4913-adc3-7497f54cd33b" Jul 06 23:25:08.780845 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:08.780819 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:08.780998 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:08.780919 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsnfg" podUID="a58729bb-f5a9-4bce-8cfc-3f98f6dd2401" Jul 06 23:25:09.781991 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:09.781520 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:09.781991 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:09.781642 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:25:10.780461 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:10.780429 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:10.780654 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:10.780549 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsnfg" podUID="a58729bb-f5a9-4bce-8cfc-3f98f6dd2401" Jul 06 23:25:10.780654 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:10.780585 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:10.780802 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:10.780662 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdnxn" podUID="f9260140-57c2-4913-adc3-7497f54cd33b" Jul 06 23:25:11.780210 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:11.780184 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:11.780590 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:11.780290 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:25:12.029840 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:12.029811 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" event={"ID":"025b4529-f609-451b-acc9-19ff4baa1378","Type":"ContainerStarted","Data":"a393bb6d5f60db669ecf552a6421cad6a7c45aa4ef7ecfa39e53d352c76a4a41"} Jul 06 23:25:12.030971 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:12.030952 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bgdhp" event={"ID":"3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5","Type":"ContainerStarted","Data":"aa7c6ad2d6fb2c9c659298394390cc2c7d665745b5064deeffd4c19bb7fc680f"} Jul 06 23:25:12.032219 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:12.032200 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal" event={"ID":"7916f8702abca842d06320c25e14c3c0","Type":"ContainerStarted","Data":"88cb232cc8a63a433dffa97f194d8d8e3e97e0a61324556bf9494532ca2bb7a6"} Jul 06 23:25:12.047607 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:12.047570 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4bk8b" podStartSLOduration=1.51963768 podStartE2EDuration="22.04755976s" totalImagesPullingTime="20.52792208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:12.047442972 +0000 UTC m=+22.761497539" watchObservedRunningTime="2026-07-06 23:25:12.04755976 +0000 UTC m=+22.761614328" Jul 06 23:25:12.078052 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:12.078012 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bgdhp" podStartSLOduration=1.407325477 podStartE2EDuration="22.077999009s" totalImagesPullingTime="20.670673532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:12.07762918 +0000 UTC m=+22.791683762" watchObservedRunningTime="2026-07-06 23:25:12.077999009 +0000 UTC m=+22.792053578" Jul 06 23:25:12.078253 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:12.078222 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-248.ec2.internal" podStartSLOduration=22.078215206 podStartE2EDuration="22.078215206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:12.061588992 +0000 UTC m=+22.775643561" watchObservedRunningTime="2026-07-06 23:25:12.078215206 +0000 UTC m=+22.792269772" Jul 06 23:25:12.780646 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:12.780518 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:12.781127 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:12.780549 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:12.781127 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:12.780765 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsnfg" podUID="a58729bb-f5a9-4bce-8cfc-3f98f6dd2401" Jul 06 23:25:12.781127 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:12.780817 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdnxn" podUID="f9260140-57c2-4913-adc3-7497f54cd33b" Jul 06 23:25:13.034756 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:13.034701 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fwj6n" event={"ID":"b2181e0c-0076-401a-a413-252ab0c6d157","Type":"ContainerStarted","Data":"16b0382c7d42ceef8729febd82d23aebf968aaa8ea678591a6723404844f721b"} Jul 06 23:25:13.038591 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:13.038572 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 06 23:25:13.038988 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:13.038962 2571 generic.go:356] "Generic (PLEG): container finished" podID="23472647-b51d-4d6b-9662-3f56bade3b4d" containerID="3b8c129438d9ac66559e0dee094656c5788295a91d47914a93c3ba244049b7a6" exitCode=1 Jul 06 23:25:13.039095 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:13.039038 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" event={"ID":"23472647-b51d-4d6b-9662-3f56bade3b4d","Type":"ContainerStarted","Data":"9b7c019517f19364d9bba4326ece95eb56628fca035f055f47ca0fd4c49f9fca"} Jul 06 23:25:13.039163 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:13.039115 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" event={"ID":"23472647-b51d-4d6b-9662-3f56bade3b4d","Type":"ContainerStarted","Data":"0e0ebca38f7ce11d438c32dd99b40773cb1fecf1387556d80ece12af5d4223ec"} Jul 06 23:25:13.039221 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:13.039167 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" event={"ID":"23472647-b51d-4d6b-9662-3f56bade3b4d","Type":"ContainerStarted","Data":"033c429adefba2ebb7ee70381f13eb7d0a755c116494e553945946e2f2ffaab3"} Jul 06 23:25:13.039269 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:13.039221 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" event={"ID":"23472647-b51d-4d6b-9662-3f56bade3b4d","Type":"ContainerStarted","Data":"46850f4dc0d47efeb19b3b38599554bd8f292c8a0dffd33fddd9cbfa17e7906e"} Jul 06 23:25:13.039320 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:13.039272 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" event={"ID":"23472647-b51d-4d6b-9662-3f56bade3b4d","Type":"ContainerDied","Data":"3b8c129438d9ac66559e0dee094656c5788295a91d47914a93c3ba244049b7a6"} Jul 06 23:25:13.039371 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:13.039323 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" event={"ID":"23472647-b51d-4d6b-9662-3f56bade3b4d","Type":"ContainerStarted","Data":"1767419196a93c38e6b60e4ed3b51acede648def622acbcec1351f6a4bef63c5"} Jul 06 23:25:13.049263 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:13.049224 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fwj6n" podStartSLOduration=2.456188022 podStartE2EDuration="23.049213226s" totalImagesPullingTime="20.593025204s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:13.048932338 +0000 UTC m=+23.762986928" watchObservedRunningTime="2026-07-06 23:25:13.049213226 +0000 UTC m=+23.763267796" Jul 06 23:25:13.780316 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:13.780143 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:13.780546 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:13.780427 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:25:14.042102 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:14.042071 2571 generic.go:356] "Generic (PLEG): container finished" podID="84e3d258c0c669908efda319c624e11f" containerID="dd3ddbb5940c2b5a04faef46979f72f96442304644b1d3a8426eda04095765ec" exitCode=0 Jul 06 23:25:14.042660 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:14.042141 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal" event={"ID":"84e3d258c0c669908efda319c624e11f","Type":"ContainerDied","Data":"dd3ddbb5940c2b5a04faef46979f72f96442304644b1d3a8426eda04095765ec"} Jul 06 23:25:14.043673 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:14.043649 2571 generic.go:356] "Generic (PLEG): container finished" podID="f1332668-fa67-4d07-9a0a-4975eadc0623" containerID="cfe7eaea51baca3e2ae8564fd1fa587786653c4c08dbf394f880dabb8272d866" exitCode=0 Jul 06 23:25:14.043803 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:14.043701 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvl5t" event={"ID":"f1332668-fa67-4d07-9a0a-4975eadc0623","Type":"ContainerDied","Data":"cfe7eaea51baca3e2ae8564fd1fa587786653c4c08dbf394f880dabb8272d866"} Jul 06 23:25:14.045123 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:14.045103 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jdmhg" event={"ID":"c92b54e3-3f51-45c9-96a7-92f26052db71","Type":"ContainerStarted","Data":"507521bc82f353f263f91a0a35a3aae7b785482312cde947742e08d6dada1c0a"} Jul 06 23:25:14.046622 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:14.046598 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kpwsm" event={"ID":"17ddd37f-3cef-44d2-990a-1ca5e4b484fc","Type":"ContainerStarted","Data":"9c237172810e811479b073d66f9481e01d7250fd408192780cb0cc97beac4fca"} Jul 06 23:25:14.048737 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:14.048659 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" event={"ID":"a181075c-e62d-4224-9cd5-582fb5e04e05","Type":"ContainerStarted","Data":"40dd0ef3c4ce7f670aa2f3566e868414287ded675b69478b62f0ca87b95502d9"} Jul 06 23:25:14.050600 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:14.050579 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l6dhh" event={"ID":"e746a856-21ba-4c28-9545-6e6d6e299f49","Type":"ContainerStarted","Data":"a455ea4fe505b6f7da4ed9173b891ce6dfa073b78abaffc1387095db014d97c7"} Jul 06 23:25:14.071592 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:14.071555 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l6dhh" podStartSLOduration=3.442562402 podStartE2EDuration="24.071543701s" totalImagesPullingTime="20.628981299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:14.071326987 +0000 UTC m=+24.785381554" watchObservedRunningTime="2026-07-06 23:25:14.071543701 +0000 UTC m=+24.785598269" Jul 06 23:25:14.100093 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:14.100058 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kpwsm" podStartSLOduration=3.49647835 podStartE2EDuration="24.100046439s" totalImagesPullingTime="20.603568089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:14.084461458 +0000 UTC m=+24.798516030" watchObservedRunningTime="2026-07-06 23:25:14.100046439 +0000 UTC m=+24.814100986" Jul 06 23:25:14.100370 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:14.100325 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jdmhg" podStartSLOduration=3.526179345 podStartE2EDuration="24.100318256s" totalImagesPullingTime="20.574138911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:14.100162512 +0000 UTC m=+24.814217063" watchObservedRunningTime="2026-07-06 23:25:14.100318256 +0000 UTC m=+24.814372826" Jul 06 23:25:14.304463 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:14.304321 2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 23:25:14.709898 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:14.709788 2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T23:25:14.304359647Z","UUID":"9575ecd4-89a2-4b30-94e8-c13e6e761bc5","Handler":null,"Name":"","Endpoint":""} Jul 06 23:25:14.711431 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:14.711409 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 23:25:14.711571 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:14.711450 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 23:25:14.780155 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:14.780125 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:14.780395 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:14.780124 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:14.780395 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:14.780237 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsnfg" podUID="a58729bb-f5a9-4bce-8cfc-3f98f6dd2401" Jul 06 23:25:14.780519 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:14.780402 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdnxn" podUID="f9260140-57c2-4913-adc3-7497f54cd33b" Jul 06 23:25:15.055121 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:15.055048 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" event={"ID":"a181075c-e62d-4224-9cd5-582fb5e04e05","Type":"ContainerStarted","Data":"0b3d8c8dec80ee71aacdd4c2113e8a46d9a69ee679683f7072981a81676c986f"} Jul 06 23:25:15.057983 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:15.057950 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal" event={"ID":"84e3d258c0c669908efda319c624e11f","Type":"ContainerStarted","Data":"761258c1985807ec30ae992f72a6c68c3161b1e986685ccec3a901bfee1e8365"} Jul 06 23:25:15.072108 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:15.072061 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-248.ec2.internal" podStartSLOduration=25.072046283 podStartE2EDuration="25.072046283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:15.071688324 +0000 UTC m=+25.785742894" watchObservedRunningTime="2026-07-06 23:25:15.072046283 +0000 UTC m=+25.786100852" Jul 06 23:25:15.780687 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:15.780663 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:15.780825 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:15.780803 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:25:16.062882 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:16.062857 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 06 23:25:16.063302 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:16.063276 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" event={"ID":"23472647-b51d-4d6b-9662-3f56bade3b4d","Type":"ContainerStarted","Data":"671508b3d9475f334b93613082a8ac0c6db711f4df252cabc058c716596dcc7c"} Jul 06 23:25:16.065622 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:16.065588 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" event={"ID":"a181075c-e62d-4224-9cd5-582fb5e04e05","Type":"ContainerStarted","Data":"29f43a3f1ea7c3969971bf5e4eb96219d5c57f21c48744c2b6414241d1ce41b2"} Jul 06 23:25:16.082502 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:16.082462 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pht6" podStartSLOduration=3.204226853 podStartE2EDuration="26.082448084s" totalImagesPullingTime="22.878221231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:24:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:16.082089282 +0000 UTC m=+26.796143853" watchObservedRunningTime="2026-07-06 23:25:16.082448084 +0000 UTC m=+26.796502652" Jul 06 23:25:16.780839 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:16.780640 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:16.780993 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:16.780652 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:16.780993 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:16.780944 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdnxn" podUID="f9260140-57c2-4913-adc3-7497f54cd33b" Jul 06 23:25:16.781078 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:16.780994 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsnfg" podUID="a58729bb-f5a9-4bce-8cfc-3f98f6dd2401" Jul 06 23:25:17.780393 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:17.780292 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:17.780798 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:17.780444 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:25:18.780864 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:18.780676 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:18.781370 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:18.780680 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:18.781370 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:18.780942 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsnfg" podUID="a58729bb-f5a9-4bce-8cfc-3f98f6dd2401" Jul 06 23:25:18.781370 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:18.781026 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdnxn" podUID="f9260140-57c2-4913-adc3-7497f54cd33b" Jul 06 23:25:18.824873 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:18.824849 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jdmhg" Jul 06 23:25:18.825387 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:18.825364 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jdmhg" Jul 06 23:25:19.073745 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:19.073619 2571 generic.go:356] "Generic (PLEG): container finished" podID="f1332668-fa67-4d07-9a0a-4975eadc0623" containerID="0b1a7e1484e5f711db67154977b152de373d077c6323b48c1711f7dc891858be" exitCode=0 Jul 06 23:25:19.073745 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:19.073710 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvl5t" event={"ID":"f1332668-fa67-4d07-9a0a-4975eadc0623","Type":"ContainerDied","Data":"0b1a7e1484e5f711db67154977b152de373d077c6323b48c1711f7dc891858be"} Jul 06 23:25:19.077111 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:19.077095 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 06 23:25:19.077506 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:19.077471 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" event={"ID":"23472647-b51d-4d6b-9662-3f56bade3b4d","Type":"ContainerStarted","Data":"907fbbe5fe6b2ddf39c042edab524847270c0f8ae8b66a25979b3ba47d30fda7"} Jul 06 23:25:19.077867 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:19.077849 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jdmhg" Jul 06 23:25:19.077997 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:19.077982 2571 scope.go:117] "RemoveContainer" containerID="3b8c129438d9ac66559e0dee094656c5788295a91d47914a93c3ba244049b7a6" Jul 06 23:25:19.078261 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:19.078243 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jdmhg" Jul 06 23:25:19.708087 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:19.708055 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:25:19.781511 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:19.781487 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:19.781839 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:19.781564 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:25:20.082052 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:20.082017 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvl5t" event={"ID":"f1332668-fa67-4d07-9a0a-4975eadc0623","Type":"ContainerStarted","Data":"df9882a6901f58425bd59b59e4fe7c57c77145835976589239596a22a654b7d0"} Jul 06 23:25:20.086534 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:20.086512 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 06 23:25:20.087130 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:20.087104 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" event={"ID":"23472647-b51d-4d6b-9662-3f56bade3b4d","Type":"ContainerStarted","Data":"d06795b91b6518cc55033961b6008d701b99bab5193c69f80b930365fa2e35bd"} Jul 06 23:25:20.087279 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:20.087265 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:25:20.087444 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:20.087432 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:25:20.104061 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:20.104042 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:25:20.105399 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:20.105380 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:25:20.161986 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:20.161949 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" podStartSLOduration=8.864457428 podStartE2EDuration="30.161936564s" totalImagesPullingTime="21.297479136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:20.160993775 +0000 UTC m=+30.875048348" watchObservedRunningTime="2026-07-06 23:25:20.161936564 +0000 UTC m=+30.875991132" Jul 06 23:25:20.379584 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:20.379555 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v8xhl"] Jul 06 23:25:20.379704 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:20.379665 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:20.379782 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:20.379749 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:25:20.382531 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:20.382512 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rsnfg"] Jul 06 23:25:20.382606 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:20.382587 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:20.382662 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:20.382647 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsnfg" podUID="a58729bb-f5a9-4bce-8cfc-3f98f6dd2401" Jul 06 23:25:20.385871 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:20.385847 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mdnxn"] Jul 06 23:25:20.386011 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:20.385999 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:20.386104 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:20.386086 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdnxn" podUID="f9260140-57c2-4913-adc3-7497f54cd33b" Jul 06 23:25:21.091700 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:21.091671 2571 generic.go:356] "Generic (PLEG): container finished" podID="f1332668-fa67-4d07-9a0a-4975eadc0623" containerID="df9882a6901f58425bd59b59e4fe7c57c77145835976589239596a22a654b7d0" exitCode=0 Jul 06 23:25:21.092131 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:21.091762 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvl5t" event={"ID":"f1332668-fa67-4d07-9a0a-4975eadc0623","Type":"ContainerDied","Data":"df9882a6901f58425bd59b59e4fe7c57c77145835976589239596a22a654b7d0"} Jul 06 23:25:21.779798 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:21.779741 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:21.779926 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:21.779840 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:25:22.096891 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:22.096748 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvl5t" event={"ID":"f1332668-fa67-4d07-9a0a-4975eadc0623","Type":"ContainerStarted","Data":"9953d0880a0e0c2df83e0a83495d64715c7af0cdf012caf87c3cf00c27d9a4b4"} Jul 06 23:25:22.405605 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:22.405532 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7xc5\" (UniqueName: \"kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5\") pod \"network-check-target-v8xhl\" (UID: \"8125ec7d-1a89-4270-af40-9b10a7f44cf5\") " pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:22.405605 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:22.405565 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret\") pod \"global-pull-secret-syncer-rsnfg\" (UID: \"a58729bb-f5a9-4bce-8cfc-3f98f6dd2401\") " pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:22.405605 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:22.405583 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs\") pod \"network-metrics-daemon-mdnxn\" (UID: \"f9260140-57c2-4913-adc3-7497f54cd33b\") " pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:22.405822 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:22.405677 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:25:22.405822 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:22.405677 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 23:25:22.405822 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:22.405693 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 23:25:22.405822 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:22.405712 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 23:25:22.405822 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:22.405738 2571 projected.go:196] Error preparing data for projected volume kube-api-access-f7xc5 for pod openshift-network-diagnostics/network-check-target-v8xhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:25:22.405822 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:22.405755 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret podName:a58729bb-f5a9-4bce-8cfc-3f98f6dd2401 nodeName:}" failed. No retries permitted until 2026-07-06 23:25:54.405737335 +0000 UTC m=+65.119791885 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret") pod "global-pull-secret-syncer-rsnfg" (UID: "a58729bb-f5a9-4bce-8cfc-3f98f6dd2401") : object "kube-system"/"original-pull-secret" not registered Jul 06 23:25:22.405822 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:22.405771 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs podName:f9260140-57c2-4913-adc3-7497f54cd33b nodeName:}" failed. No retries permitted until 2026-07-06 23:25:54.405763117 +0000 UTC m=+65.119817666 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs") pod "network-metrics-daemon-mdnxn" (UID: "f9260140-57c2-4913-adc3-7497f54cd33b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 23:25:22.405822 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:22.405784 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5 podName:8125ec7d-1a89-4270-af40-9b10a7f44cf5 nodeName:}" failed. No retries permitted until 2026-07-06 23:25:54.405776812 +0000 UTC m=+65.119831360 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f7xc5" (UniqueName: "kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5") pod "network-check-target-v8xhl" (UID: "8125ec7d-1a89-4270-af40-9b10a7f44cf5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 23:25:22.780415 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:22.780383 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:22.780415 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:22.780403 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:22.780656 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:22.780486 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsnfg" podUID="a58729bb-f5a9-4bce-8cfc-3f98f6dd2401" Jul 06 23:25:22.780656 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:22.780633 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdnxn" podUID="f9260140-57c2-4913-adc3-7497f54cd33b" Jul 06 23:25:23.101210 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:23.101130 2571 generic.go:356] "Generic (PLEG): container finished" podID="f1332668-fa67-4d07-9a0a-4975eadc0623" containerID="9953d0880a0e0c2df83e0a83495d64715c7af0cdf012caf87c3cf00c27d9a4b4" exitCode=0 Jul 06 23:25:23.101210 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:23.101195 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvl5t" event={"ID":"f1332668-fa67-4d07-9a0a-4975eadc0623","Type":"ContainerDied","Data":"9953d0880a0e0c2df83e0a83495d64715c7af0cdf012caf87c3cf00c27d9a4b4"} Jul 06 23:25:23.779860 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:23.779828 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:23.780001 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:23.779963 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:25:24.780784 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:24.780753 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:24.781172 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:24.780851 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsnfg" podUID="a58729bb-f5a9-4bce-8cfc-3f98f6dd2401" Jul 06 23:25:24.781172 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:24.781012 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:24.781172 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:24.781116 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdnxn" podUID="f9260140-57c2-4913-adc3-7497f54cd33b" Jul 06 23:25:25.780402 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:25.780367 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:25.780563 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:25.780505 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v8xhl" podUID="8125ec7d-1a89-4270-af40-9b10a7f44cf5" Jul 06 23:25:26.034578 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.034550 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-248.ec2.internal" event="NodeReady" Jul 06 23:25:26.034983 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.034691 2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 23:25:26.082039 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.082004 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-58kg2"] Jul 06 23:25:26.084585 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.084558 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b4s7d"] Jul 06 23:25:26.084752 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.084708 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:26.087011 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.086392 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b4s7d" Jul 06 23:25:26.087362 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.087323 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 23:25:26.087362 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.087358 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8bk56\"" Jul 06 23:25:26.087530 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.087358 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 23:25:26.088642 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.088628 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wfrjk\"" Jul 06 23:25:26.088888 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.088872 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 23:25:26.088971 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.088949 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 23:25:26.089042 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.088874 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 23:25:26.095432 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.095413 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b4s7d"] Jul 06 23:25:26.096153 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.096130 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-58kg2"] Jul 06 23:25:26.230790 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.230761 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a157e85-8951-46be-bbef-ea3583fc73aa-metrics-tls\") pod \"dns-default-58kg2\" (UID: \"1a157e85-8951-46be-bbef-ea3583fc73aa\") " pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:26.230956 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.230800 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a157e85-8951-46be-bbef-ea3583fc73aa-tmp-dir\") pod \"dns-default-58kg2\" (UID: \"1a157e85-8951-46be-bbef-ea3583fc73aa\") " pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:26.230956 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.230816 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a06f7512-2cd6-402b-a4eb-c85052df5dbd-cert\") pod \"ingress-canary-b4s7d\" (UID: \"a06f7512-2cd6-402b-a4eb-c85052df5dbd\") " pod="openshift-ingress-canary/ingress-canary-b4s7d" Jul 06 23:25:26.230956 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.230907 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z4gl\" (UniqueName: \"kubernetes.io/projected/a06f7512-2cd6-402b-a4eb-c85052df5dbd-kube-api-access-7z4gl\") pod \"ingress-canary-b4s7d\" (UID: \"a06f7512-2cd6-402b-a4eb-c85052df5dbd\") " pod="openshift-ingress-canary/ingress-canary-b4s7d" Jul 06 23:25:26.231079 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.231020 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz5ss\" (UniqueName: \"kubernetes.io/projected/1a157e85-8951-46be-bbef-ea3583fc73aa-kube-api-access-fz5ss\") pod \"dns-default-58kg2\" (UID: \"1a157e85-8951-46be-bbef-ea3583fc73aa\") " pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:26.231079 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.231060 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a157e85-8951-46be-bbef-ea3583fc73aa-config-volume\") pod \"dns-default-58kg2\" (UID: \"1a157e85-8951-46be-bbef-ea3583fc73aa\") " pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:26.332213 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.332129 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a157e85-8951-46be-bbef-ea3583fc73aa-metrics-tls\") pod \"dns-default-58kg2\" (UID: \"1a157e85-8951-46be-bbef-ea3583fc73aa\") " pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:26.332213 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.332184 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a157e85-8951-46be-bbef-ea3583fc73aa-tmp-dir\") pod \"dns-default-58kg2\" (UID: \"1a157e85-8951-46be-bbef-ea3583fc73aa\") " pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:26.332213 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.332207 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a06f7512-2cd6-402b-a4eb-c85052df5dbd-cert\") pod \"ingress-canary-b4s7d\" (UID: \"a06f7512-2cd6-402b-a4eb-c85052df5dbd\") " pod="openshift-ingress-canary/ingress-canary-b4s7d" Jul 06 23:25:26.332489 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.332243 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4gl\" (UniqueName: \"kubernetes.io/projected/a06f7512-2cd6-402b-a4eb-c85052df5dbd-kube-api-access-7z4gl\") pod \"ingress-canary-b4s7d\" (UID: \"a06f7512-2cd6-402b-a4eb-c85052df5dbd\") " pod="openshift-ingress-canary/ingress-canary-b4s7d" Jul 06 23:25:26.332489 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:26.332284 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:25:26.332489 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.332392 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz5ss\" (UniqueName: \"kubernetes.io/projected/1a157e85-8951-46be-bbef-ea3583fc73aa-kube-api-access-fz5ss\") pod \"dns-default-58kg2\" (UID: \"1a157e85-8951-46be-bbef-ea3583fc73aa\") " pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:26.332489 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.332440 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a157e85-8951-46be-bbef-ea3583fc73aa-config-volume\") pod \"dns-default-58kg2\" (UID: \"1a157e85-8951-46be-bbef-ea3583fc73aa\") " pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:26.332659 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:26.332521 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a157e85-8951-46be-bbef-ea3583fc73aa-metrics-tls podName:1a157e85-8951-46be-bbef-ea3583fc73aa nodeName:}" failed. No retries permitted until 2026-07-06 23:25:26.832495407 +0000 UTC m=+37.546549956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a157e85-8951-46be-bbef-ea3583fc73aa-metrics-tls") pod "dns-default-58kg2" (UID: "1a157e85-8951-46be-bbef-ea3583fc73aa") : secret "dns-default-metrics-tls" not found Jul 06 23:25:26.332659 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.332592 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a157e85-8951-46be-bbef-ea3583fc73aa-tmp-dir\") pod \"dns-default-58kg2\" (UID: \"1a157e85-8951-46be-bbef-ea3583fc73aa\") " pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:26.333030 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.333013 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a157e85-8951-46be-bbef-ea3583fc73aa-config-volume\") pod \"dns-default-58kg2\" (UID: \"1a157e85-8951-46be-bbef-ea3583fc73aa\") " pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:26.337047 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.337028 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a06f7512-2cd6-402b-a4eb-c85052df5dbd-cert\") pod \"ingress-canary-b4s7d\" (UID: \"a06f7512-2cd6-402b-a4eb-c85052df5dbd\") " pod="openshift-ingress-canary/ingress-canary-b4s7d" Jul 06 23:25:26.340808 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.340787 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz5ss\" (UniqueName: \"kubernetes.io/projected/1a157e85-8951-46be-bbef-ea3583fc73aa-kube-api-access-fz5ss\") pod \"dns-default-58kg2\" (UID: \"1a157e85-8951-46be-bbef-ea3583fc73aa\") " pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:26.341036 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.341002 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z4gl\" (UniqueName: \"kubernetes.io/projected/a06f7512-2cd6-402b-a4eb-c85052df5dbd-kube-api-access-7z4gl\") pod \"ingress-canary-b4s7d\" (UID: \"a06f7512-2cd6-402b-a4eb-c85052df5dbd\") " pod="openshift-ingress-canary/ingress-canary-b4s7d" Jul 06 23:25:26.403814 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.403779 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b4s7d" Jul 06 23:25:26.780712 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.780686 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:26.780894 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.780687 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:26.785488 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.785468 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 23:25:26.785591 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.785517 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 23:25:26.785652 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.785616 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vsplq\"" Jul 06 23:25:26.836307 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:26.836264 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a157e85-8951-46be-bbef-ea3583fc73aa-metrics-tls\") pod \"dns-default-58kg2\" (UID: \"1a157e85-8951-46be-bbef-ea3583fc73aa\") " pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:26.836440 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:26.836425 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:25:26.836508 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:26.836498 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a157e85-8951-46be-bbef-ea3583fc73aa-metrics-tls podName:1a157e85-8951-46be-bbef-ea3583fc73aa nodeName:}" failed. No retries permitted until 2026-07-06 23:25:27.836479117 +0000 UTC m=+38.550533666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a157e85-8951-46be-bbef-ea3583fc73aa-metrics-tls") pod "dns-default-58kg2" (UID: "1a157e85-8951-46be-bbef-ea3583fc73aa") : secret "dns-default-metrics-tls" not found Jul 06 23:25:27.045597 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:27.045401 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b4s7d"] Jul 06 23:25:27.049543 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:25:27.049519 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06f7512_2cd6_402b_a4eb_c85052df5dbd.slice/crio-e0170ca20312bd354387489c7bb496733714bbf02e58d6a7e7ef12f1ff8058dd WatchSource:0}: Error finding container e0170ca20312bd354387489c7bb496733714bbf02e58d6a7e7ef12f1ff8058dd: Status 404 returned error can't find the container with id e0170ca20312bd354387489c7bb496733714bbf02e58d6a7e7ef12f1ff8058dd Jul 06 23:25:27.113667 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:27.113635 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvl5t" event={"ID":"f1332668-fa67-4d07-9a0a-4975eadc0623","Type":"ContainerStarted","Data":"21327d3488b5a89fd60d7c1779c813750f6f6f877ebf094c7cca92403f731ad1"} Jul 06 23:25:27.114919 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:27.114893 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b4s7d" event={"ID":"a06f7512-2cd6-402b-a4eb-c85052df5dbd","Type":"ContainerStarted","Data":"e0170ca20312bd354387489c7bb496733714bbf02e58d6a7e7ef12f1ff8058dd"} Jul 06 23:25:27.779879 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:27.779844 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:27.782815 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:27.782791 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 23:25:27.783468 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:27.783444 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9hjn2\"" Jul 06 23:25:27.783468 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:27.783449 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 23:25:27.843120 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:27.843084 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a157e85-8951-46be-bbef-ea3583fc73aa-metrics-tls\") pod \"dns-default-58kg2\" (UID: \"1a157e85-8951-46be-bbef-ea3583fc73aa\") " pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:27.843285 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:27.843219 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:25:27.843285 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:27.843284 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a157e85-8951-46be-bbef-ea3583fc73aa-metrics-tls podName:1a157e85-8951-46be-bbef-ea3583fc73aa nodeName:}" failed. No retries permitted until 2026-07-06 23:25:29.843262429 +0000 UTC m=+40.557316990 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a157e85-8951-46be-bbef-ea3583fc73aa-metrics-tls") pod "dns-default-58kg2" (UID: "1a157e85-8951-46be-bbef-ea3583fc73aa") : secret "dns-default-metrics-tls" not found Jul 06 23:25:28.121068 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:28.120986 2571 generic.go:356] "Generic (PLEG): container finished" podID="f1332668-fa67-4d07-9a0a-4975eadc0623" containerID="21327d3488b5a89fd60d7c1779c813750f6f6f877ebf094c7cca92403f731ad1" exitCode=0 Jul 06 23:25:28.121491 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:28.121069 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvl5t" event={"ID":"f1332668-fa67-4d07-9a0a-4975eadc0623","Type":"ContainerDied","Data":"21327d3488b5a89fd60d7c1779c813750f6f6f877ebf094c7cca92403f731ad1"} Jul 06 23:25:29.126652 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:29.126616 2571 generic.go:356] "Generic (PLEG): container finished" podID="f1332668-fa67-4d07-9a0a-4975eadc0623" containerID="1eb5dcb4609b90c8786e038a0a1c6b812907b890737ff617546bed7fdcaa9354" exitCode=0 Jul 06 23:25:29.127051 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:29.126694 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvl5t" event={"ID":"f1332668-fa67-4d07-9a0a-4975eadc0623","Type":"ContainerDied","Data":"1eb5dcb4609b90c8786e038a0a1c6b812907b890737ff617546bed7fdcaa9354"} Jul 06 23:25:29.857390 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:29.857354 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a157e85-8951-46be-bbef-ea3583fc73aa-metrics-tls\") pod \"dns-default-58kg2\" (UID: \"1a157e85-8951-46be-bbef-ea3583fc73aa\") " pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:29.857561 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:29.857486 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:25:29.857561 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:29.857542 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a157e85-8951-46be-bbef-ea3583fc73aa-metrics-tls podName:1a157e85-8951-46be-bbef-ea3583fc73aa nodeName:}" failed. No retries permitted until 2026-07-06 23:25:33.857527381 +0000 UTC m=+44.571581930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a157e85-8951-46be-bbef-ea3583fc73aa-metrics-tls") pod "dns-default-58kg2" (UID: "1a157e85-8951-46be-bbef-ea3583fc73aa") : secret "dns-default-metrics-tls" not found Jul 06 23:25:30.130890 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:30.130665 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b4s7d" event={"ID":"a06f7512-2cd6-402b-a4eb-c85052df5dbd","Type":"ContainerStarted","Data":"655479de7cd23ab763d2840a6d647ef8650915aa90b987584bbc59f5ec3fb45e"} Jul 06 23:25:30.133996 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:30.133970 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvl5t" event={"ID":"f1332668-fa67-4d07-9a0a-4975eadc0623","Type":"ContainerStarted","Data":"b0f165813bd78718fd7805f98165da1cf2c9f70333542d03dc98466bdd5da73f"} Jul 06 23:25:30.146943 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:30.146899 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b4s7d" podStartSLOduration=1.7188717580000001 podStartE2EDuration="4.146883642s" totalImagesPullingTime="2.428011884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:30.145169296 +0000 UTC m=+40.859223859" watchObservedRunningTime="2026-07-06 23:25:30.146883642 +0000 UTC m=+40.860938210" Jul 06 23:25:30.171170 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:30.170948 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bvl5t" podStartSLOduration=9.657070335 podStartE2EDuration="40.170931121s" totalImagesPullingTime="30.513860786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:24:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:30.167911729 +0000 UTC m=+40.881966300" watchObservedRunningTime="2026-07-06 23:25:30.170931121 +0000 UTC m=+40.884985691" Jul 06 23:25:31.922047 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:31.922020 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-frnkj"] Jul 06 23:25:31.955271 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:31.955245 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-frnkj"] Jul 06 23:25:31.955379 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:31.955340 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:31.957546 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:31.957520 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 23:25:31.957667 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:31.957611 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 23:25:31.957788 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:31.957774 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d7jfq\"" Jul 06 23:25:31.958424 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:31.958408 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 23:25:31.958511 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:31.958486 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 23:25:31.971081 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:31.971034 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c162094-42a9-4902-a4ea-0b8ab848b05d-crio-socket\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:31.971200 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:31.971122 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c162094-42a9-4902-a4ea-0b8ab848b05d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:31.971200 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:31.971158 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c162094-42a9-4902-a4ea-0b8ab848b05d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:31.971200 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:31.971182 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-998nn\" (UniqueName: \"kubernetes.io/projected/2c162094-42a9-4902-a4ea-0b8ab848b05d-kube-api-access-998nn\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:31.971360 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:31.971239 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c162094-42a9-4902-a4ea-0b8ab848b05d-data-volume\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:32.071988 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.071961 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c162094-42a9-4902-a4ea-0b8ab848b05d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:32.072134 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.071991 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c162094-42a9-4902-a4ea-0b8ab848b05d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:32.072134 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.072008 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-998nn\" (UniqueName: \"kubernetes.io/projected/2c162094-42a9-4902-a4ea-0b8ab848b05d-kube-api-access-998nn\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:32.072134 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.072033 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c162094-42a9-4902-a4ea-0b8ab848b05d-data-volume\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:32.072134 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.072053 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c162094-42a9-4902-a4ea-0b8ab848b05d-crio-socket\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:32.072134 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:32.072075 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 23:25:32.072134 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:32.072115 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:25:32.072435 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:32.072165 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c162094-42a9-4902-a4ea-0b8ab848b05d-kube-rbac-proxy-cm podName:2c162094-42a9-4902-a4ea-0b8ab848b05d nodeName:}" failed. No retries permitted until 2026-07-06 23:25:32.572145357 +0000 UTC m=+43.286199905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c162094-42a9-4902-a4ea-0b8ab848b05d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-frnkj" (UID: "2c162094-42a9-4902-a4ea-0b8ab848b05d") : configmap "kube-rbac-proxy" not found Jul 06 23:25:32.072435 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:32.072184 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c162094-42a9-4902-a4ea-0b8ab848b05d-insights-runtime-extractor-tls podName:2c162094-42a9-4902-a4ea-0b8ab848b05d nodeName:}" failed. No retries permitted until 2026-07-06 23:25:32.572175191 +0000 UTC m=+43.286229741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c162094-42a9-4902-a4ea-0b8ab848b05d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-frnkj" (UID: "2c162094-42a9-4902-a4ea-0b8ab848b05d") : secret "insights-runtime-extractor-tls" not found Jul 06 23:25:32.072435 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.072295 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c162094-42a9-4902-a4ea-0b8ab848b05d-crio-socket\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:32.072435 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.072318 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c162094-42a9-4902-a4ea-0b8ab848b05d-data-volume\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:32.089330 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.089299 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-998nn\" (UniqueName: \"kubernetes.io/projected/2c162094-42a9-4902-a4ea-0b8ab848b05d-kube-api-access-998nn\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:32.256484 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.256431 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-rkhp5"] Jul 06 23:25:32.277138 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.277114 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-rkhp5"] Jul 06 23:25:32.277224 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.277203 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-rkhp5" Jul 06 23:25:32.279603 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.279580 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 06 23:25:32.279699 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.279676 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-gwsmw\"" Jul 06 23:25:32.279857 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.279842 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 06 23:25:32.374178 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.374154 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wnw6\" (UniqueName: \"kubernetes.io/projected/e4edfb48-79ac-4ffa-a279-96c5aa1a8348-kube-api-access-7wnw6\") pod \"migrator-d57d58cc9-rkhp5\" (UID: \"e4edfb48-79ac-4ffa-a279-96c5aa1a8348\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-rkhp5" Jul 06 23:25:32.475358 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.475332 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wnw6\" (UniqueName: \"kubernetes.io/projected/e4edfb48-79ac-4ffa-a279-96c5aa1a8348-kube-api-access-7wnw6\") pod \"migrator-d57d58cc9-rkhp5\" (UID: \"e4edfb48-79ac-4ffa-a279-96c5aa1a8348\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-rkhp5" Jul 06 23:25:32.485636 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.485612 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wnw6\" (UniqueName: \"kubernetes.io/projected/e4edfb48-79ac-4ffa-a279-96c5aa1a8348-kube-api-access-7wnw6\") pod \"migrator-d57d58cc9-rkhp5\" (UID: \"e4edfb48-79ac-4ffa-a279-96c5aa1a8348\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-rkhp5" Jul 06 23:25:32.575736 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.575655 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c162094-42a9-4902-a4ea-0b8ab848b05d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:32.575736 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.575694 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c162094-42a9-4902-a4ea-0b8ab848b05d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:32.575890 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:32.575786 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 23:25:32.575890 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:32.575841 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c162094-42a9-4902-a4ea-0b8ab848b05d-kube-rbac-proxy-cm podName:2c162094-42a9-4902-a4ea-0b8ab848b05d nodeName:}" failed. No retries permitted until 2026-07-06 23:25:33.575827334 +0000 UTC m=+44.289881882 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c162094-42a9-4902-a4ea-0b8ab848b05d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-frnkj" (UID: "2c162094-42a9-4902-a4ea-0b8ab848b05d") : configmap "kube-rbac-proxy" not found Jul 06 23:25:32.575890 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:32.575840 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:25:32.576007 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:32.575895 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c162094-42a9-4902-a4ea-0b8ab848b05d-insights-runtime-extractor-tls podName:2c162094-42a9-4902-a4ea-0b8ab848b05d nodeName:}" failed. No retries permitted until 2026-07-06 23:25:33.575880222 +0000 UTC m=+44.289934783 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c162094-42a9-4902-a4ea-0b8ab848b05d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-frnkj" (UID: "2c162094-42a9-4902-a4ea-0b8ab848b05d") : secret "insights-runtime-extractor-tls" not found Jul 06 23:25:32.585270 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.585254 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-rkhp5" Jul 06 23:25:32.722810 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:32.722773 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-rkhp5"] Jul 06 23:25:32.726517 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:25:32.726494 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4edfb48_79ac_4ffa_a279_96c5aa1a8348.slice/crio-4d2297eec48fe725d16f36ffe8e865855f7546f25c23422ca82e83fff95e548f WatchSource:0}: Error finding container 4d2297eec48fe725d16f36ffe8e865855f7546f25c23422ca82e83fff95e548f: Status 404 returned error can't find the container with id 4d2297eec48fe725d16f36ffe8e865855f7546f25c23422ca82e83fff95e548f Jul 06 23:25:33.142080 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.142045 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-rkhp5" event={"ID":"e4edfb48-79ac-4ffa-a279-96c5aa1a8348","Type":"ContainerStarted","Data":"4d2297eec48fe725d16f36ffe8e865855f7546f25c23422ca82e83fff95e548f"} Jul 06 23:25:33.586368 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.586329 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c162094-42a9-4902-a4ea-0b8ab848b05d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:33.586368 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.586370 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c162094-42a9-4902-a4ea-0b8ab848b05d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:33.586573 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:33.586453 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 23:25:33.586573 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:33.586470 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:25:33.586573 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:33.586533 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c162094-42a9-4902-a4ea-0b8ab848b05d-insights-runtime-extractor-tls podName:2c162094-42a9-4902-a4ea-0b8ab848b05d nodeName:}" failed. No retries permitted until 2026-07-06 23:25:35.586518266 +0000 UTC m=+46.300572827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c162094-42a9-4902-a4ea-0b8ab848b05d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-frnkj" (UID: "2c162094-42a9-4902-a4ea-0b8ab848b05d") : secret "insights-runtime-extractor-tls" not found Jul 06 23:25:33.586573 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:33.586552 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c162094-42a9-4902-a4ea-0b8ab848b05d-kube-rbac-proxy-cm podName:2c162094-42a9-4902-a4ea-0b8ab848b05d nodeName:}" failed. No retries permitted until 2026-07-06 23:25:35.586537342 +0000 UTC m=+46.300591892 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c162094-42a9-4902-a4ea-0b8ab848b05d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-frnkj" (UID: "2c162094-42a9-4902-a4ea-0b8ab848b05d") : configmap "kube-rbac-proxy" not found Jul 06 23:25:33.708406 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.708373 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-d5qvf"] Jul 06 23:25:33.728844 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.728818 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-d5qvf"] Jul 06 23:25:33.728981 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.728924 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-d5qvf" Jul 06 23:25:33.731426 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.731403 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 06 23:25:33.731521 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.731478 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 06 23:25:33.731521 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.731480 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 06 23:25:33.732235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.732213 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lj9sc\"" Jul 06 23:25:33.732350 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.732248 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 06 23:25:33.788000 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.787968 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/99ff79c8-3c29-4f80-9afd-e9db8fd15ae0-signing-cabundle\") pod \"service-ca-758f599c5-d5qvf\" (UID: \"99ff79c8-3c29-4f80-9afd-e9db8fd15ae0\") " pod="openshift-service-ca/service-ca-758f599c5-d5qvf" Jul 06 23:25:33.788157 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.788032 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/99ff79c8-3c29-4f80-9afd-e9db8fd15ae0-signing-key\") pod \"service-ca-758f599c5-d5qvf\" (UID: \"99ff79c8-3c29-4f80-9afd-e9db8fd15ae0\") " pod="openshift-service-ca/service-ca-758f599c5-d5qvf" Jul 06 23:25:33.788157 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.788066 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhzvs\" (UniqueName: \"kubernetes.io/projected/99ff79c8-3c29-4f80-9afd-e9db8fd15ae0-kube-api-access-qhzvs\") pod \"service-ca-758f599c5-d5qvf\" (UID: \"99ff79c8-3c29-4f80-9afd-e9db8fd15ae0\") " pod="openshift-service-ca/service-ca-758f599c5-d5qvf" Jul 06 23:25:33.888658 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.888575 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/99ff79c8-3c29-4f80-9afd-e9db8fd15ae0-signing-cabundle\") pod \"service-ca-758f599c5-d5qvf\" (UID: \"99ff79c8-3c29-4f80-9afd-e9db8fd15ae0\") " pod="openshift-service-ca/service-ca-758f599c5-d5qvf" Jul 06 23:25:33.888658 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.888633 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a157e85-8951-46be-bbef-ea3583fc73aa-metrics-tls\") pod \"dns-default-58kg2\" (UID: \"1a157e85-8951-46be-bbef-ea3583fc73aa\") " pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:33.888897 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:33.888748 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 23:25:33.888897 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:33.888809 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a157e85-8951-46be-bbef-ea3583fc73aa-metrics-tls podName:1a157e85-8951-46be-bbef-ea3583fc73aa nodeName:}" failed. No retries permitted until 2026-07-06 23:25:41.888792186 +0000 UTC m=+52.602846739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a157e85-8951-46be-bbef-ea3583fc73aa-metrics-tls") pod "dns-default-58kg2" (UID: "1a157e85-8951-46be-bbef-ea3583fc73aa") : secret "dns-default-metrics-tls" not found Jul 06 23:25:33.888897 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.888802 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/99ff79c8-3c29-4f80-9afd-e9db8fd15ae0-signing-key\") pod \"service-ca-758f599c5-d5qvf\" (UID: \"99ff79c8-3c29-4f80-9afd-e9db8fd15ae0\") " pod="openshift-service-ca/service-ca-758f599c5-d5qvf" Jul 06 23:25:33.888897 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.888842 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhzvs\" (UniqueName: \"kubernetes.io/projected/99ff79c8-3c29-4f80-9afd-e9db8fd15ae0-kube-api-access-qhzvs\") pod \"service-ca-758f599c5-d5qvf\" (UID: \"99ff79c8-3c29-4f80-9afd-e9db8fd15ae0\") " pod="openshift-service-ca/service-ca-758f599c5-d5qvf" Jul 06 23:25:33.889409 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.889366 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/99ff79c8-3c29-4f80-9afd-e9db8fd15ae0-signing-cabundle\") pod \"service-ca-758f599c5-d5qvf\" (UID: \"99ff79c8-3c29-4f80-9afd-e9db8fd15ae0\") " pod="openshift-service-ca/service-ca-758f599c5-d5qvf" Jul 06 23:25:33.891562 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.891535 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/99ff79c8-3c29-4f80-9afd-e9db8fd15ae0-signing-key\") pod \"service-ca-758f599c5-d5qvf\" (UID: \"99ff79c8-3c29-4f80-9afd-e9db8fd15ae0\") " pod="openshift-service-ca/service-ca-758f599c5-d5qvf" Jul 06 23:25:33.898308 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:33.898282 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhzvs\" (UniqueName: \"kubernetes.io/projected/99ff79c8-3c29-4f80-9afd-e9db8fd15ae0-kube-api-access-qhzvs\") pod \"service-ca-758f599c5-d5qvf\" (UID: \"99ff79c8-3c29-4f80-9afd-e9db8fd15ae0\") " pod="openshift-service-ca/service-ca-758f599c5-d5qvf" Jul 06 23:25:34.037676 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:34.037644 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-d5qvf" Jul 06 23:25:34.103922 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:34.103891 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l6dhh_e746a856-21ba-4c28-9545-6e6d6e299f49/dns-node-resolver/0.log" Jul 06 23:25:34.187170 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:34.187141 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-d5qvf"] Jul 06 23:25:34.190838 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:25:34.190804 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ff79c8_3c29_4f80_9afd_e9db8fd15ae0.slice/crio-516dad5cb0a69ca392af024ed0e20f5e83eba6bf4484da97b547f829705fbd87 WatchSource:0}: Error finding container 516dad5cb0a69ca392af024ed0e20f5e83eba6bf4484da97b547f829705fbd87: Status 404 returned error can't find the container with id 516dad5cb0a69ca392af024ed0e20f5e83eba6bf4484da97b547f829705fbd87 Jul 06 23:25:34.703060 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:34.703035 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kpwsm_17ddd37f-3cef-44d2-990a-1ca5e4b484fc/node-ca/0.log" Jul 06 23:25:35.148512 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:35.148433 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-rkhp5" event={"ID":"e4edfb48-79ac-4ffa-a279-96c5aa1a8348","Type":"ContainerStarted","Data":"1c338e65c7064e40195ed3430b4cd3597293490c20223fbcf7b5e1917d76e2ad"} Jul 06 23:25:35.148512 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:35.148489 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-rkhp5" event={"ID":"e4edfb48-79ac-4ffa-a279-96c5aa1a8348","Type":"ContainerStarted","Data":"a5dacc0e137f9a6cad0a4b9a9ae8a7c45f599b6d80f42e93885a8f63453b96eb"} Jul 06 23:25:35.149455 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:35.149434 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-d5qvf" event={"ID":"99ff79c8-3c29-4f80-9afd-e9db8fd15ae0","Type":"ContainerStarted","Data":"516dad5cb0a69ca392af024ed0e20f5e83eba6bf4484da97b547f829705fbd87"} Jul 06 23:25:35.166100 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:35.166057 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-rkhp5" podStartSLOduration=1.497250055 podStartE2EDuration="3.166043041s" totalImagesPullingTime="1.668792986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:25:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:35.165091502 +0000 UTC m=+45.879146073" watchObservedRunningTime="2026-07-06 23:25:35.166043041 +0000 UTC m=+45.880097589" Jul 06 23:25:35.599006 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:35.598970 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c162094-42a9-4902-a4ea-0b8ab848b05d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:35.599405 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:35.599014 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c162094-42a9-4902-a4ea-0b8ab848b05d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:35.599405 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:35.599101 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 23:25:35.599405 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:35.599139 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:25:35.599405 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:35.599169 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c162094-42a9-4902-a4ea-0b8ab848b05d-kube-rbac-proxy-cm podName:2c162094-42a9-4902-a4ea-0b8ab848b05d nodeName:}" failed. No retries permitted until 2026-07-06 23:25:39.599151527 +0000 UTC m=+50.313206075 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c162094-42a9-4902-a4ea-0b8ab848b05d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-frnkj" (UID: "2c162094-42a9-4902-a4ea-0b8ab848b05d") : configmap "kube-rbac-proxy" not found Jul 06 23:25:35.599405 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:35.599186 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c162094-42a9-4902-a4ea-0b8ab848b05d-insights-runtime-extractor-tls podName:2c162094-42a9-4902-a4ea-0b8ab848b05d nodeName:}" failed. No retries permitted until 2026-07-06 23:25:39.599178515 +0000 UTC m=+50.313233063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c162094-42a9-4902-a4ea-0b8ab848b05d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-frnkj" (UID: "2c162094-42a9-4902-a4ea-0b8ab848b05d") : secret "insights-runtime-extractor-tls" not found Jul 06 23:25:35.702664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:35.702633 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b4s7d_a06f7512-2cd6-402b-a4eb-c85052df5dbd/serve-healthcheck-canary/0.log" Jul 06 23:25:37.154936 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:37.154899 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-d5qvf" event={"ID":"99ff79c8-3c29-4f80-9afd-e9db8fd15ae0","Type":"ContainerStarted","Data":"f815672c62bdc15c4af911a7cf281bf36ff83c4e46e8c020e89dc88d2529e59b"} Jul 06 23:25:37.169261 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:37.169211 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-d5qvf" podStartSLOduration=1.9630169020000001 podStartE2EDuration="4.169194386s" totalImagesPullingTime="2.206177484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:37.169111581 +0000 UTC m=+47.883166152" watchObservedRunningTime="2026-07-06 23:25:37.169194386 +0000 UTC m=+47.883248960" Jul 06 23:25:39.627059 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:39.627025 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c162094-42a9-4902-a4ea-0b8ab848b05d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:39.627059 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:39.627061 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c162094-42a9-4902-a4ea-0b8ab848b05d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:39.627455 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:39.627168 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 23:25:39.627455 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:25:39.627222 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c162094-42a9-4902-a4ea-0b8ab848b05d-insights-runtime-extractor-tls podName:2c162094-42a9-4902-a4ea-0b8ab848b05d nodeName:}" failed. No retries permitted until 2026-07-06 23:25:47.627207116 +0000 UTC m=+58.341261665 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c162094-42a9-4902-a4ea-0b8ab848b05d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-frnkj" (UID: "2c162094-42a9-4902-a4ea-0b8ab848b05d") : secret "insights-runtime-extractor-tls" not found Jul 06 23:25:39.638398 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:39.638360 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c162094-42a9-4902-a4ea-0b8ab848b05d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:41.943991 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:41.943953 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a157e85-8951-46be-bbef-ea3583fc73aa-metrics-tls\") pod \"dns-default-58kg2\" (UID: \"1a157e85-8951-46be-bbef-ea3583fc73aa\") " pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:41.946527 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:41.946503 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a157e85-8951-46be-bbef-ea3583fc73aa-metrics-tls\") pod \"dns-default-58kg2\" (UID: \"1a157e85-8951-46be-bbef-ea3583fc73aa\") " pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:41.997928 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:41.997909 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:42.132908 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:42.132881 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-58kg2"] Jul 06 23:25:42.135998 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:25:42.135975 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a157e85_8951_46be_bbef_ea3583fc73aa.slice/crio-a212be7451e6ee626f8ff2e3c784ecb6ae6e77d085d452be50c868b7fb2548d2 WatchSource:0}: Error finding container a212be7451e6ee626f8ff2e3c784ecb6ae6e77d085d452be50c868b7fb2548d2: Status 404 returned error can't find the container with id a212be7451e6ee626f8ff2e3c784ecb6ae6e77d085d452be50c868b7fb2548d2 Jul 06 23:25:42.169381 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:42.169349 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-58kg2" event={"ID":"1a157e85-8951-46be-bbef-ea3583fc73aa","Type":"ContainerStarted","Data":"a212be7451e6ee626f8ff2e3c784ecb6ae6e77d085d452be50c868b7fb2548d2"} Jul 06 23:25:44.177247 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:44.177214 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-58kg2" event={"ID":"1a157e85-8951-46be-bbef-ea3583fc73aa","Type":"ContainerStarted","Data":"5a36d44e8854e87f8bde617fd6f1251d9be075286714fe1fc6df4c788880f00b"} Jul 06 23:25:44.177705 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:44.177262 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-58kg2" event={"ID":"1a157e85-8951-46be-bbef-ea3583fc73aa","Type":"ContainerStarted","Data":"eb8d7cc2eead4c5165abea60511dbaee7f45bc3d7b608c36dc8dc56f82f7b92f"} Jul 06 23:25:44.177705 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:44.177352 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:44.193473 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:44.193413 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-58kg2" podStartSLOduration=16.481270947 podStartE2EDuration="18.193397545s" totalImagesPullingTime="1.712126598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:44.192771589 +0000 UTC m=+54.906826159" watchObservedRunningTime="2026-07-06 23:25:44.193397545 +0000 UTC m=+54.907452110" Jul 06 23:25:47.686085 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:47.686051 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c162094-42a9-4902-a4ea-0b8ab848b05d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:47.688431 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:47.688409 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c162094-42a9-4902-a4ea-0b8ab848b05d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-frnkj\" (UID: \"2c162094-42a9-4902-a4ea-0b8ab848b05d\") " pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:47.863917 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:47.863884 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-frnkj" Jul 06 23:25:47.983048 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:47.983022 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-frnkj"] Jul 06 23:25:47.986162 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:25:47.986131 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c162094_42a9_4902_a4ea_0b8ab848b05d.slice/crio-e727931120fcc3669e9b71780df82ead23e7ca0729566a5fbf7bd63d54ed6f37 WatchSource:0}: Error finding container e727931120fcc3669e9b71780df82ead23e7ca0729566a5fbf7bd63d54ed6f37: Status 404 returned error can't find the container with id e727931120fcc3669e9b71780df82ead23e7ca0729566a5fbf7bd63d54ed6f37 Jul 06 23:25:48.192588 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:48.192526 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-frnkj" event={"ID":"2c162094-42a9-4902-a4ea-0b8ab848b05d","Type":"ContainerStarted","Data":"e727931120fcc3669e9b71780df82ead23e7ca0729566a5fbf7bd63d54ed6f37"} Jul 06 23:25:52.107198 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:52.107122 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c7gg2" Jul 06 23:25:52.205624 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:52.205593 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-frnkj" event={"ID":"2c162094-42a9-4902-a4ea-0b8ab848b05d","Type":"ContainerStarted","Data":"a0689c2681db257f82c29aeb41ad04e91140aeda7c0ea732c5391d7a211851fe"} Jul 06 23:25:53.210949 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:53.210914 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-frnkj" event={"ID":"2c162094-42a9-4902-a4ea-0b8ab848b05d","Type":"ContainerStarted","Data":"ec2d5366f800dd53b58e6d29501708593bb730624d543612aa8382489e6de032"} Jul 06 23:25:54.183652 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:54.183618 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-58kg2" Jul 06 23:25:54.434274 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:54.434188 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7xc5\" (UniqueName: \"kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5\") pod \"network-check-target-v8xhl\" (UID: \"8125ec7d-1a89-4270-af40-9b10a7f44cf5\") " pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:54.434274 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:54.434233 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret\") pod \"global-pull-secret-syncer-rsnfg\" (UID: \"a58729bb-f5a9-4bce-8cfc-3f98f6dd2401\") " pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:54.434765 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:54.434309 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs\") pod \"network-metrics-daemon-mdnxn\" (UID: \"f9260140-57c2-4913-adc3-7497f54cd33b\") " pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:54.437546 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:54.437510 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 23:25:54.437672 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:54.437653 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 23:25:54.438751 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:54.438706 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 23:25:54.447050 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:54.447023 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a58729bb-f5a9-4bce-8cfc-3f98f6dd2401-original-pull-secret\") pod \"global-pull-secret-syncer-rsnfg\" (UID: \"a58729bb-f5a9-4bce-8cfc-3f98f6dd2401\") " pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:54.447145 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:54.447077 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9260140-57c2-4913-adc3-7497f54cd33b-metrics-certs\") pod \"network-metrics-daemon-mdnxn\" (UID: \"f9260140-57c2-4913-adc3-7497f54cd33b\") " pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:54.447145 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:54.447139 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 23:25:54.457676 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:54.457654 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7xc5\" (UniqueName: \"kubernetes.io/projected/8125ec7d-1a89-4270-af40-9b10a7f44cf5-kube-api-access-f7xc5\") pod \"network-check-target-v8xhl\" (UID: \"8125ec7d-1a89-4270-af40-9b10a7f44cf5\") " pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:54.493204 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:54.493177 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9hjn2\"" Jul 06 23:25:54.500753 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:54.500712 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:25:54.692031 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:54.691935 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsnfg" Jul 06 23:25:54.700955 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:54.700935 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vsplq\"" Jul 06 23:25:54.709031 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:54.709012 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdnxn" Jul 06 23:25:54.947265 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:54.947209 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mdnxn"] Jul 06 23:25:54.950473 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:25:54.950450 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9260140_57c2_4913_adc3_7497f54cd33b.slice/crio-dd9c9aac50ad91e07cebc9a08c019a9bf23b35779c3867dd279d45f4fbe7c273 WatchSource:0}: Error finding container dd9c9aac50ad91e07cebc9a08c019a9bf23b35779c3867dd279d45f4fbe7c273: Status 404 returned error can't find the container with id dd9c9aac50ad91e07cebc9a08c019a9bf23b35779c3867dd279d45f4fbe7c273 Jul 06 23:25:54.979315 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:54.979290 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rsnfg"] Jul 06 23:25:54.982173 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:25:54.982151 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58729bb_f5a9_4bce_8cfc_3f98f6dd2401.slice/crio-4a027ffbab28761e4cd68bf5f10e737f9fd12c21f407a6ab0a91f006fac4ee07 WatchSource:0}: Error finding container 4a027ffbab28761e4cd68bf5f10e737f9fd12c21f407a6ab0a91f006fac4ee07: Status 404 returned error can't find the container with id 4a027ffbab28761e4cd68bf5f10e737f9fd12c21f407a6ab0a91f006fac4ee07 Jul 06 23:25:55.146855 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:55.146823 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v8xhl"] Jul 06 23:25:55.149532 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:25:55.149508 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8125ec7d_1a89_4270_af40_9b10a7f44cf5.slice/crio-b0e79daa0eb084ddf8ea069a9514c3befa9ec78f53bbf3786c0ac65d5658940b WatchSource:0}: Error finding container b0e79daa0eb084ddf8ea069a9514c3befa9ec78f53bbf3786c0ac65d5658940b: Status 404 returned error can't find the container with id b0e79daa0eb084ddf8ea069a9514c3befa9ec78f53bbf3786c0ac65d5658940b Jul 06 23:25:55.218308 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:55.218215 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rsnfg" event={"ID":"a58729bb-f5a9-4bce-8cfc-3f98f6dd2401","Type":"ContainerStarted","Data":"4a027ffbab28761e4cd68bf5f10e737f9fd12c21f407a6ab0a91f006fac4ee07"} Jul 06 23:25:55.220113 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:55.220077 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-frnkj" event={"ID":"2c162094-42a9-4902-a4ea-0b8ab848b05d","Type":"ContainerStarted","Data":"60bdac8099f7583c24d5d26876512b48b7ee8925e9af6893200b223de2b689c3"} Jul 06 23:25:55.221305 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:55.221281 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mdnxn" event={"ID":"f9260140-57c2-4913-adc3-7497f54cd33b","Type":"ContainerStarted","Data":"dd9c9aac50ad91e07cebc9a08c019a9bf23b35779c3867dd279d45f4fbe7c273"} Jul 06 23:25:55.222262 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:55.222243 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v8xhl" event={"ID":"8125ec7d-1a89-4270-af40-9b10a7f44cf5","Type":"ContainerStarted","Data":"b0e79daa0eb084ddf8ea069a9514c3befa9ec78f53bbf3786c0ac65d5658940b"} Jul 06 23:25:55.237377 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:55.237334 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-frnkj" podStartSLOduration=17.484690366 podStartE2EDuration="24.237320748s" totalImagesPullingTime="6.752630382s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:25:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:55.236969349 +0000 UTC m=+65.951023920" watchObservedRunningTime="2026-07-06 23:25:55.237320748 +0000 UTC m=+65.951375318" Jul 06 23:25:55.956571 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:55.956538 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln"] Jul 06 23:25:55.961927 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:55.961906 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:55.966339 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:55.965959 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 06 23:25:55.966339 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:55.966021 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 23:25:55.966339 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:55.966165 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 23:25:55.967122 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:55.967095 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 06 23:25:55.967395 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:55.967371 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 06 23:25:55.967586 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:55.967567 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 23:25:55.967801 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:55.967784 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 06 23:25:55.978958 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:55.978932 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln"] Jul 06 23:25:56.048701 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.048483 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/44122ead-dc6b-454a-b6a0-cd532843a263-ca\") pod \"cluster-proxy-proxy-agent-cc49d9b4b-ttwln\" (UID: \"44122ead-dc6b-454a-b6a0-cd532843a263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.048701 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.048520 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/44122ead-dc6b-454a-b6a0-cd532843a263-hub\") pod \"cluster-proxy-proxy-agent-cc49d9b4b-ttwln\" (UID: \"44122ead-dc6b-454a-b6a0-cd532843a263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.048701 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.048541 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44122ead-dc6b-454a-b6a0-cd532843a263-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cc49d9b4b-ttwln\" (UID: \"44122ead-dc6b-454a-b6a0-cd532843a263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.048701 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.048566 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/44122ead-dc6b-454a-b6a0-cd532843a263-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cc49d9b4b-ttwln\" (UID: \"44122ead-dc6b-454a-b6a0-cd532843a263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.048701 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.048596 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/44122ead-dc6b-454a-b6a0-cd532843a263-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cc49d9b4b-ttwln\" (UID: \"44122ead-dc6b-454a-b6a0-cd532843a263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.048701 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.048618 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn94c\" (UniqueName: \"kubernetes.io/projected/44122ead-dc6b-454a-b6a0-cd532843a263-kube-api-access-cn94c\") pod \"cluster-proxy-proxy-agent-cc49d9b4b-ttwln\" (UID: \"44122ead-dc6b-454a-b6a0-cd532843a263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.151584 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.149287 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/44122ead-dc6b-454a-b6a0-cd532843a263-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cc49d9b4b-ttwln\" (UID: \"44122ead-dc6b-454a-b6a0-cd532843a263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.151584 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.149337 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn94c\" (UniqueName: \"kubernetes.io/projected/44122ead-dc6b-454a-b6a0-cd532843a263-kube-api-access-cn94c\") pod \"cluster-proxy-proxy-agent-cc49d9b4b-ttwln\" (UID: \"44122ead-dc6b-454a-b6a0-cd532843a263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.151584 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.149395 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/44122ead-dc6b-454a-b6a0-cd532843a263-ca\") pod \"cluster-proxy-proxy-agent-cc49d9b4b-ttwln\" (UID: \"44122ead-dc6b-454a-b6a0-cd532843a263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.151584 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.149427 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/44122ead-dc6b-454a-b6a0-cd532843a263-hub\") pod \"cluster-proxy-proxy-agent-cc49d9b4b-ttwln\" (UID: \"44122ead-dc6b-454a-b6a0-cd532843a263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.151584 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.149452 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44122ead-dc6b-454a-b6a0-cd532843a263-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cc49d9b4b-ttwln\" (UID: \"44122ead-dc6b-454a-b6a0-cd532843a263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.151584 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.149479 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/44122ead-dc6b-454a-b6a0-cd532843a263-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cc49d9b4b-ttwln\" (UID: \"44122ead-dc6b-454a-b6a0-cd532843a263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.151584 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.151248 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/44122ead-dc6b-454a-b6a0-cd532843a263-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cc49d9b4b-ttwln\" (UID: \"44122ead-dc6b-454a-b6a0-cd532843a263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.155423 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.154585 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/44122ead-dc6b-454a-b6a0-cd532843a263-ca\") pod \"cluster-proxy-proxy-agent-cc49d9b4b-ttwln\" (UID: \"44122ead-dc6b-454a-b6a0-cd532843a263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.163845 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.156979 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44122ead-dc6b-454a-b6a0-cd532843a263-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cc49d9b4b-ttwln\" (UID: \"44122ead-dc6b-454a-b6a0-cd532843a263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.163845 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.157652 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/44122ead-dc6b-454a-b6a0-cd532843a263-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cc49d9b4b-ttwln\" (UID: \"44122ead-dc6b-454a-b6a0-cd532843a263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.163845 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.160850 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/44122ead-dc6b-454a-b6a0-cd532843a263-hub\") pod \"cluster-proxy-proxy-agent-cc49d9b4b-ttwln\" (UID: \"44122ead-dc6b-454a-b6a0-cd532843a263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.163845 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.162099 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn94c\" (UniqueName: \"kubernetes.io/projected/44122ead-dc6b-454a-b6a0-cd532843a263-kube-api-access-cn94c\") pod \"cluster-proxy-proxy-agent-cc49d9b4b-ttwln\" (UID: \"44122ead-dc6b-454a-b6a0-cd532843a263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.228987 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.228941 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mdnxn" event={"ID":"f9260140-57c2-4913-adc3-7497f54cd33b","Type":"ContainerStarted","Data":"020c8d4b3d218d48f3d53df8a1c2b0024b60d5545c38b804c876d5f421a2c1ea"} Jul 06 23:25:56.229159 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.229015 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mdnxn" event={"ID":"f9260140-57c2-4913-adc3-7497f54cd33b","Type":"ContainerStarted","Data":"d3efca3ab964e845e943edbf903f262735071804ceea971d5a0bb0c499cef458"} Jul 06 23:25:56.261078 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.261012 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mdnxn" podStartSLOduration=65.298159952 podStartE2EDuration="1m6.260990303s" totalImagesPullingTime="962.830351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:56.259567393 +0000 UTC m=+66.973621961" watchObservedRunningTime="2026-07-06 23:25:56.260990303 +0000 UTC m=+66.975044877" Jul 06 23:25:56.285747 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.285700 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" Jul 06 23:25:56.450408 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:56.450332 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln"] Jul 06 23:25:56.454009 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:25:56.453976 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44122ead_dc6b_454a_b6a0_cd532843a263.slice/crio-a99e76130713bc837747b94b4dd9ba4c5bcd2dffdb13c41fb99f97ec76a4fdb9 WatchSource:0}: Error finding container a99e76130713bc837747b94b4dd9ba4c5bcd2dffdb13c41fb99f97ec76a4fdb9: Status 404 returned error can't find the container with id a99e76130713bc837747b94b4dd9ba4c5bcd2dffdb13c41fb99f97ec76a4fdb9 Jul 06 23:25:57.237802 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:25:57.237710 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" event={"ID":"44122ead-dc6b-454a-b6a0-cd532843a263","Type":"ContainerStarted","Data":"a99e76130713bc837747b94b4dd9ba4c5bcd2dffdb13c41fb99f97ec76a4fdb9"} Jul 06 23:26:02.258612 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:02.258573 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rsnfg" event={"ID":"a58729bb-f5a9-4bce-8cfc-3f98f6dd2401","Type":"ContainerStarted","Data":"4004782be7da9be151fcb441b8d6bb333ed896256beeb16f9e0c290d93f01c96"} Jul 06 23:26:02.260948 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:02.260920 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" event={"ID":"44122ead-dc6b-454a-b6a0-cd532843a263","Type":"ContainerStarted","Data":"389ac2d54b69a2b02446016a369f1b79a06f3b997d82c13d9289682ac64cf27d"} Jul 06 23:26:02.261066 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:02.260962 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" event={"ID":"44122ead-dc6b-454a-b6a0-cd532843a263","Type":"ContainerStarted","Data":"f621e28d47bcd4d0e41706f4a79171bf1e475e6dfb69c51ea948ab14141e1daa"} Jul 06 23:26:02.261066 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:02.260980 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" event={"ID":"44122ead-dc6b-454a-b6a0-cd532843a263","Type":"ContainerStarted","Data":"67c433ff45c30d94bd1c2dacbbdef0bdd77d8c173e86141406de8d7ce5ee76a4"} Jul 06 23:26:02.262607 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:02.262583 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v8xhl" event={"ID":"8125ec7d-1a89-4270-af40-9b10a7f44cf5","Type":"ContainerStarted","Data":"582a1c9f523957e70bc828f26d9fbdebeb6c0668c38120a4208fbcf648e86739"} Jul 06 23:26:02.262765 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:02.262747 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:26:02.274004 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:02.273954 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rsnfg" podStartSLOduration=65.2827324 podStartE2EDuration="1m12.273940568s" totalImagesPullingTime="6.991208168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:26:02.273031767 +0000 UTC m=+72.987086338" watchObservedRunningTime="2026-07-06 23:26:02.273940568 +0000 UTC m=+72.987995138" Jul 06 23:26:02.294108 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:02.294055 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc49d9b4b-ttwln" podStartSLOduration=1.716327292 podStartE2EDuration="7.294040445s" totalImagesPullingTime="5.577713153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:26:02.292809436 +0000 UTC m=+73.006864005" watchObservedRunningTime="2026-07-06 23:26:02.294040445 +0000 UTC m=+73.008095015" Jul 06 23:26:02.308068 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:02.308017 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v8xhl" podStartSLOduration=65.498430209 podStartE2EDuration="1m12.308001702s" totalImagesPullingTime="6.809571493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:26:02.307890812 +0000 UTC m=+73.021945382" watchObservedRunningTime="2026-07-06 23:26:02.308001702 +0000 UTC m=+73.022056273" Jul 06 23:26:07.886327 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.886293 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj"] Jul 06 23:26:07.892304 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.891750 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:07.895363 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.895335 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hfr7m"] Jul 06 23:26:07.898255 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.897134 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 06 23:26:07.898255 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.897152 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 23:26:07.898255 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.897191 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 06 23:26:07.898255 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.897326 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 23:26:07.898255 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.898188 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 23:26:07.898255 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.898195 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 06 23:26:07.898617 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.898450 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-s6gqh\"" Jul 06 23:26:07.900138 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.900110 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:07.903150 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.902976 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 23:26:07.903150 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.903093 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 23:26:07.903150 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.903149 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 23:26:07.903394 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.903371 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-d69jr\"" Jul 06 23:26:07.917205 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.917122 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj"] Jul 06 23:26:07.933237 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.933208 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ae132236-cbea-4271-b751-535163747780-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-wxmwj\" (UID: \"ae132236-cbea-4271-b751-535163747780\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:07.933384 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.933254 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae132236-cbea-4271-b751-535163747780-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-wxmwj\" (UID: \"ae132236-cbea-4271-b751-535163747780\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:07.933384 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.933274 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae132236-cbea-4271-b751-535163747780-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-wxmwj\" (UID: \"ae132236-cbea-4271-b751-535163747780\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:07.933384 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.933303 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae132236-cbea-4271-b751-535163747780-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-wxmwj\" (UID: \"ae132236-cbea-4271-b751-535163747780\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:07.933384 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.933355 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ae132236-cbea-4271-b751-535163747780-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-wxmwj\" (UID: \"ae132236-cbea-4271-b751-535163747780\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:07.933538 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:07.933410 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfbdz\" (UniqueName: \"kubernetes.io/projected/ae132236-cbea-4271-b751-535163747780-kube-api-access-dfbdz\") pod \"kube-state-metrics-7c9dbb8b7d-wxmwj\" (UID: \"ae132236-cbea-4271-b751-535163747780\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:08.034198 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.034162 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxnbl\" (UniqueName: \"kubernetes.io/projected/8195eebd-4c62-4a63-8d35-d05df15f75d9-kube-api-access-cxnbl\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.034198 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.034199 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8195eebd-4c62-4a63-8d35-d05df15f75d9-node-exporter-wtmp\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.034449 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.034229 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfbdz\" (UniqueName: \"kubernetes.io/projected/ae132236-cbea-4271-b751-535163747780-kube-api-access-dfbdz\") pod \"kube-state-metrics-7c9dbb8b7d-wxmwj\" (UID: \"ae132236-cbea-4271-b751-535163747780\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:08.034449 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.034281 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ae132236-cbea-4271-b751-535163747780-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-wxmwj\" (UID: \"ae132236-cbea-4271-b751-535163747780\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:08.034449 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.034311 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae132236-cbea-4271-b751-535163747780-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-wxmwj\" (UID: \"ae132236-cbea-4271-b751-535163747780\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:08.034630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.034445 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae132236-cbea-4271-b751-535163747780-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-wxmwj\" (UID: \"ae132236-cbea-4271-b751-535163747780\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:08.035135 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.035101 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8195eebd-4c62-4a63-8d35-d05df15f75d9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.035352 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.035327 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8195eebd-4c62-4a63-8d35-d05df15f75d9-node-exporter-accelerators-collector-config\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.035471 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.035180 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae132236-cbea-4271-b751-535163747780-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-wxmwj\" (UID: \"ae132236-cbea-4271-b751-535163747780\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:08.035471 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.035393 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae132236-cbea-4271-b751-535163747780-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-wxmwj\" (UID: \"ae132236-cbea-4271-b751-535163747780\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:08.035471 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.035186 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ae132236-cbea-4271-b751-535163747780-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-wxmwj\" (UID: \"ae132236-cbea-4271-b751-535163747780\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:08.035471 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.035427 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8195eebd-4c62-4a63-8d35-d05df15f75d9-metrics-client-ca\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.035674 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.035542 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8195eebd-4c62-4a63-8d35-d05df15f75d9-node-exporter-tls\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.035674 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.035571 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8195eebd-4c62-4a63-8d35-d05df15f75d9-node-exporter-textfile\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.035674 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.035603 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ae132236-cbea-4271-b751-535163747780-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-wxmwj\" (UID: \"ae132236-cbea-4271-b751-535163747780\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:08.035674 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.035640 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8195eebd-4c62-4a63-8d35-d05df15f75d9-sys\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.035674 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.035666 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8195eebd-4c62-4a63-8d35-d05df15f75d9-root\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.035999 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.035978 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ae132236-cbea-4271-b751-535163747780-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-wxmwj\" (UID: \"ae132236-cbea-4271-b751-535163747780\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:08.037158 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.037130 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae132236-cbea-4271-b751-535163747780-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-wxmwj\" (UID: \"ae132236-cbea-4271-b751-535163747780\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:08.038109 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.038090 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae132236-cbea-4271-b751-535163747780-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-wxmwj\" (UID: \"ae132236-cbea-4271-b751-535163747780\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:08.049766 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.049711 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfbdz\" (UniqueName: \"kubernetes.io/projected/ae132236-cbea-4271-b751-535163747780-kube-api-access-dfbdz\") pod \"kube-state-metrics-7c9dbb8b7d-wxmwj\" (UID: \"ae132236-cbea-4271-b751-535163747780\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:08.136889 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.136802 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxnbl\" (UniqueName: \"kubernetes.io/projected/8195eebd-4c62-4a63-8d35-d05df15f75d9-kube-api-access-cxnbl\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.136889 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.136844 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8195eebd-4c62-4a63-8d35-d05df15f75d9-node-exporter-wtmp\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.136889 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.136886 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8195eebd-4c62-4a63-8d35-d05df15f75d9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.137103 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.136918 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8195eebd-4c62-4a63-8d35-d05df15f75d9-node-exporter-accelerators-collector-config\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.137103 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.136942 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8195eebd-4c62-4a63-8d35-d05df15f75d9-metrics-client-ca\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.137103 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.136965 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8195eebd-4c62-4a63-8d35-d05df15f75d9-node-exporter-tls\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.137103 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.136985 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8195eebd-4c62-4a63-8d35-d05df15f75d9-node-exporter-textfile\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.137103 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.137024 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8195eebd-4c62-4a63-8d35-d05df15f75d9-node-exporter-wtmp\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.137103 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.137077 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8195eebd-4c62-4a63-8d35-d05df15f75d9-sys\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.137103 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.137021 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8195eebd-4c62-4a63-8d35-d05df15f75d9-sys\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.137334 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.137112 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8195eebd-4c62-4a63-8d35-d05df15f75d9-root\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.137334 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.137174 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8195eebd-4c62-4a63-8d35-d05df15f75d9-root\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.137619 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.137589 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8195eebd-4c62-4a63-8d35-d05df15f75d9-node-exporter-textfile\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.137769 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.137684 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8195eebd-4c62-4a63-8d35-d05df15f75d9-node-exporter-accelerators-collector-config\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.137970 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.137952 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8195eebd-4c62-4a63-8d35-d05df15f75d9-metrics-client-ca\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.140096 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.140074 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8195eebd-4c62-4a63-8d35-d05df15f75d9-node-exporter-tls\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.140184 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.140116 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8195eebd-4c62-4a63-8d35-d05df15f75d9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.158610 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.158584 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxnbl\" (UniqueName: \"kubernetes.io/projected/8195eebd-4c62-4a63-8d35-d05df15f75d9-kube-api-access-cxnbl\") pod \"node-exporter-hfr7m\" (UID: \"8195eebd-4c62-4a63-8d35-d05df15f75d9\") " pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.207283 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.207253 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" Jul 06 23:26:08.213047 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.213022 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hfr7m" Jul 06 23:26:08.225251 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:26:08.225225 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8195eebd_4c62_4a63_8d35_d05df15f75d9.slice/crio-30e7f3ec65d4bf15faddf976745c6e2dde3f6949285fa861f1a55584fb5e5e20 WatchSource:0}: Error finding container 30e7f3ec65d4bf15faddf976745c6e2dde3f6949285fa861f1a55584fb5e5e20: Status 404 returned error can't find the container with id 30e7f3ec65d4bf15faddf976745c6e2dde3f6949285fa861f1a55584fb5e5e20 Jul 06 23:26:08.284144 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.284049 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hfr7m" event={"ID":"8195eebd-4c62-4a63-8d35-d05df15f75d9","Type":"ContainerStarted","Data":"30e7f3ec65d4bf15faddf976745c6e2dde3f6949285fa861f1a55584fb5e5e20"} Jul 06 23:26:08.350695 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:08.350665 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj"] Jul 06 23:26:08.353695 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:26:08.353672 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae132236_cbea_4271_b751_535163747780.slice/crio-4ba00ad4f70e04809e532310e2fa147d94a0bd32d976410f870bf7756512abbd WatchSource:0}: Error finding container 4ba00ad4f70e04809e532310e2fa147d94a0bd32d976410f870bf7756512abbd: Status 404 returned error can't find the container with id 4ba00ad4f70e04809e532310e2fa147d94a0bd32d976410f870bf7756512abbd Jul 06 23:26:09.297332 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:09.297298 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" event={"ID":"ae132236-cbea-4271-b751-535163747780","Type":"ContainerStarted","Data":"4ba00ad4f70e04809e532310e2fa147d94a0bd32d976410f870bf7756512abbd"} Jul 06 23:26:10.302944 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:10.302814 2571 generic.go:356] "Generic (PLEG): container finished" podID="8195eebd-4c62-4a63-8d35-d05df15f75d9" containerID="b9b6b629360278263448ad130aa51fff5707ef01bfb8696c6507b31731b688e9" exitCode=0 Jul 06 23:26:10.302944 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:10.302899 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hfr7m" event={"ID":"8195eebd-4c62-4a63-8d35-d05df15f75d9","Type":"ContainerDied","Data":"b9b6b629360278263448ad130aa51fff5707ef01bfb8696c6507b31731b688e9"} Jul 06 23:26:10.305259 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:10.305233 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" event={"ID":"ae132236-cbea-4271-b751-535163747780","Type":"ContainerStarted","Data":"0f86a5dd1e2b209caadfb45499ef34d0fa990d09f71ef9ddeee54d897f60c6cb"} Jul 06 23:26:10.305353 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:10.305275 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" event={"ID":"ae132236-cbea-4271-b751-535163747780","Type":"ContainerStarted","Data":"cac8703c50358486719c9202d46b9ce06d864d1bc34b6de1584fd5466c0b968d"} Jul 06 23:26:10.305353 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:10.305294 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" event={"ID":"ae132236-cbea-4271-b751-535163747780","Type":"ContainerStarted","Data":"c855a259fa7f45c0fa7ed2aa048719fa25a40c8b0216efc27fa2740e81bc23cf"} Jul 06 23:26:10.339739 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:10.339681 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-wxmwj" podStartSLOduration=2.10403711 podStartE2EDuration="3.339668331s" totalImagesPullingTime="1.235631221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:26:10.338535626 +0000 UTC m=+81.052590222" watchObservedRunningTime="2026-07-06 23:26:10.339668331 +0000 UTC m=+81.053722902" Jul 06 23:26:11.310185 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:11.310137 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hfr7m" event={"ID":"8195eebd-4c62-4a63-8d35-d05df15f75d9","Type":"ContainerStarted","Data":"18bb3455ad929832e02eb5c635ab68b0b5ad8bc5ac95fe11b9635bbdaaad41e3"} Jul 06 23:26:11.310619 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:11.310203 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hfr7m" event={"ID":"8195eebd-4c62-4a63-8d35-d05df15f75d9","Type":"ContainerStarted","Data":"836e38ceded381f6c4b7d096c693fc321f20ad5c9a25bb395178a85b97cd877c"} Jul 06 23:26:11.328934 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:11.328890 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hfr7m" podStartSLOduration=3.255042395 podStartE2EDuration="4.328870839s" totalImagesPullingTime="997.709673ms" totalInitContainerRuntime="76.118771ms" isStatefulPod=true podCreationTimestamp="2026-07-06 23:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:26:11.327583579 +0000 UTC m=+82.041638151" watchObservedRunningTime="2026-07-06 23:26:11.328870839 +0000 UTC m=+82.042925408" Jul 06 23:26:12.589868 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:12.589834 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-kr2lr"] Jul 06 23:26:12.593150 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:12.593134 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kr2lr" Jul 06 23:26:12.595343 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:12.595325 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 23:26:12.595439 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:12.595413 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ls9g9\"" Jul 06 23:26:12.600535 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:12.600510 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-kr2lr"] Jul 06 23:26:12.670160 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:12.670127 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fd890264-dc09-4401-ab1a-6d3b37fb5bde-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-kr2lr\" (UID: \"fd890264-dc09-4401-ab1a-6d3b37fb5bde\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kr2lr" Jul 06 23:26:12.670330 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:12.670166 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdktv\" (UniqueName: \"kubernetes.io/projected/fd890264-dc09-4401-ab1a-6d3b37fb5bde-kube-api-access-mdktv\") pod \"monitoring-plugin-d5dcff7cf-kr2lr\" (UID: \"fd890264-dc09-4401-ab1a-6d3b37fb5bde\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kr2lr" Jul 06 23:26:12.771441 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:12.771396 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fd890264-dc09-4401-ab1a-6d3b37fb5bde-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-kr2lr\" (UID: \"fd890264-dc09-4401-ab1a-6d3b37fb5bde\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kr2lr" Jul 06 23:26:12.771663 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:12.771474 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdktv\" (UniqueName: \"kubernetes.io/projected/fd890264-dc09-4401-ab1a-6d3b37fb5bde-kube-api-access-mdktv\") pod \"monitoring-plugin-d5dcff7cf-kr2lr\" (UID: \"fd890264-dc09-4401-ab1a-6d3b37fb5bde\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kr2lr" Jul 06 23:26:12.774061 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:12.774025 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fd890264-dc09-4401-ab1a-6d3b37fb5bde-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-kr2lr\" (UID: \"fd890264-dc09-4401-ab1a-6d3b37fb5bde\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kr2lr" Jul 06 23:26:12.779928 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:12.779906 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdktv\" (UniqueName: \"kubernetes.io/projected/fd890264-dc09-4401-ab1a-6d3b37fb5bde-kube-api-access-mdktv\") pod \"monitoring-plugin-d5dcff7cf-kr2lr\" (UID: \"fd890264-dc09-4401-ab1a-6d3b37fb5bde\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kr2lr" Jul 06 23:26:12.902570 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:12.902474 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kr2lr" Jul 06 23:26:13.026491 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:13.026454 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-kr2lr"] Jul 06 23:26:13.030474 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:26:13.030448 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd890264_dc09_4401_ab1a_6d3b37fb5bde.slice/crio-bf2738f541495b388a1e6c625861c4a628c1635ac1cdd1942638ea8af34796d5 WatchSource:0}: Error finding container bf2738f541495b388a1e6c625861c4a628c1635ac1cdd1942638ea8af34796d5: Status 404 returned error can't find the container with id bf2738f541495b388a1e6c625861c4a628c1635ac1cdd1942638ea8af34796d5 Jul 06 23:26:13.318369 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:13.318335 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kr2lr" event={"ID":"fd890264-dc09-4401-ab1a-6d3b37fb5bde","Type":"ContainerStarted","Data":"bf2738f541495b388a1e6c625861c4a628c1635ac1cdd1942638ea8af34796d5"} Jul 06 23:26:14.323595 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:14.323552 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kr2lr" event={"ID":"fd890264-dc09-4401-ab1a-6d3b37fb5bde","Type":"ContainerStarted","Data":"43a77c4de506d1bbc57ad366069ee8613e802f98db750951d4d0a2de4d513ce1"} Jul 06 23:26:14.324004 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:14.323826 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kr2lr" Jul 06 23:26:14.328640 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:14.328620 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kr2lr" Jul 06 23:26:14.342396 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:14.342352 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kr2lr" podStartSLOduration=1.180382066 podStartE2EDuration="2.342337384s" totalImagesPullingTime="1.161955318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:26:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:26:14.34133968 +0000 UTC m=+85.055394251" watchObservedRunningTime="2026-07-06 23:26:14.342337384 +0000 UTC m=+85.056391961" Jul 06 23:26:33.268539 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:26:33.268501 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v8xhl" Jul 06 23:29:37.031809 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:37.031774 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4"] Jul 06 23:29:37.035049 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:37.035029 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" Jul 06 23:29:37.037930 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:37.037912 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 23:29:37.038265 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:37.038242 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cf5fj\"" Jul 06 23:29:37.038876 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:37.038860 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 23:29:37.057401 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:37.057374 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4"] Jul 06 23:29:37.198812 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:37.198781 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk27r\" (UniqueName: \"kubernetes.io/projected/9395f48f-945c-46e5-8c16-6d254bcc9e82-kube-api-access-rk27r\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4\" (UID: \"9395f48f-945c-46e5-8c16-6d254bcc9e82\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" Jul 06 23:29:37.198995 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:37.198824 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9395f48f-945c-46e5-8c16-6d254bcc9e82-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4\" (UID: \"9395f48f-945c-46e5-8c16-6d254bcc9e82\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" Jul 06 23:29:37.198995 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:37.198847 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9395f48f-945c-46e5-8c16-6d254bcc9e82-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4\" (UID: \"9395f48f-945c-46e5-8c16-6d254bcc9e82\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" Jul 06 23:29:37.300085 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:37.299986 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk27r\" (UniqueName: \"kubernetes.io/projected/9395f48f-945c-46e5-8c16-6d254bcc9e82-kube-api-access-rk27r\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4\" (UID: \"9395f48f-945c-46e5-8c16-6d254bcc9e82\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" Jul 06 23:29:37.300085 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:37.300047 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9395f48f-945c-46e5-8c16-6d254bcc9e82-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4\" (UID: \"9395f48f-945c-46e5-8c16-6d254bcc9e82\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" Jul 06 23:29:37.300085 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:37.300071 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9395f48f-945c-46e5-8c16-6d254bcc9e82-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4\" (UID: \"9395f48f-945c-46e5-8c16-6d254bcc9e82\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" Jul 06 23:29:37.300496 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:37.300470 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9395f48f-945c-46e5-8c16-6d254bcc9e82-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4\" (UID: \"9395f48f-945c-46e5-8c16-6d254bcc9e82\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" Jul 06 23:29:37.300553 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:37.300480 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9395f48f-945c-46e5-8c16-6d254bcc9e82-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4\" (UID: \"9395f48f-945c-46e5-8c16-6d254bcc9e82\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" Jul 06 23:29:37.310215 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:37.310184 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk27r\" (UniqueName: \"kubernetes.io/projected/9395f48f-945c-46e5-8c16-6d254bcc9e82-kube-api-access-rk27r\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4\" (UID: \"9395f48f-945c-46e5-8c16-6d254bcc9e82\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" Jul 06 23:29:37.344214 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:37.344183 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" Jul 06 23:29:37.470384 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:37.470357 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4"] Jul 06 23:29:37.473006 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:29:37.472978 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9395f48f_945c_46e5_8c16_6d254bcc9e82.slice/crio-e303ffb3e0ad57f7a9f0d14bd55a363bbff359d53566ba74e1a99d1b1a9c3c88 WatchSource:0}: Error finding container e303ffb3e0ad57f7a9f0d14bd55a363bbff359d53566ba74e1a99d1b1a9c3c88: Status 404 returned error can't find the container with id e303ffb3e0ad57f7a9f0d14bd55a363bbff359d53566ba74e1a99d1b1a9c3c88 Jul 06 23:29:37.983294 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:37.983245 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" event={"ID":"9395f48f-945c-46e5-8c16-6d254bcc9e82","Type":"ContainerStarted","Data":"e303ffb3e0ad57f7a9f0d14bd55a363bbff359d53566ba74e1a99d1b1a9c3c88"} Jul 06 23:29:44.005964 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:44.005928 2571 generic.go:356] "Generic (PLEG): container finished" podID="9395f48f-945c-46e5-8c16-6d254bcc9e82" containerID="cbfb5fa11da85351a05d95fba4ac3c70660927c5b007abce58191d39ed51777b" exitCode=0 Jul 06 23:29:44.006411 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:44.005975 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" event={"ID":"9395f48f-945c-46e5-8c16-6d254bcc9e82","Type":"ContainerDied","Data":"cbfb5fa11da85351a05d95fba4ac3c70660927c5b007abce58191d39ed51777b"} Jul 06 23:29:47.018696 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:47.018664 2571 generic.go:356] "Generic (PLEG): container finished" podID="9395f48f-945c-46e5-8c16-6d254bcc9e82" containerID="2a5000d6bdfd14b3bfcc7b7f67ffe0cb4cec644af4ce34711d477042889c1321" exitCode=0 Jul 06 23:29:47.019092 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:47.018739 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" event={"ID":"9395f48f-945c-46e5-8c16-6d254bcc9e82","Type":"ContainerDied","Data":"2a5000d6bdfd14b3bfcc7b7f67ffe0cb4cec644af4ce34711d477042889c1321"} Jul 06 23:29:49.650317 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:49.650282 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 06 23:29:49.655682 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:49.655649 2571 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 23:29:49.655835 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:49.655790 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 06 23:29:54.046897 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:54.046861 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" event={"ID":"9395f48f-945c-46e5-8c16-6d254bcc9e82","Type":"ContainerStarted","Data":"c1fb67760aec00b0504bbaf4f79b487ea2537071203196f8bf09761662e1c237"} Jul 06 23:29:54.068490 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:54.068425 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" podStartSLOduration=2.180814685 podStartE2EDuration="17.068403221s" totalImagesPullingTime="14.887588536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:29:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:29:54.067356365 +0000 UTC m=+304.781410935" watchObservedRunningTime="2026-07-06 23:29:54.068403221 +0000 UTC m=+304.782457836" Jul 06 23:29:55.052883 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:55.052845 2571 generic.go:356] "Generic (PLEG): container finished" podID="9395f48f-945c-46e5-8c16-6d254bcc9e82" containerID="c1fb67760aec00b0504bbaf4f79b487ea2537071203196f8bf09761662e1c237" exitCode=0 Jul 06 23:29:55.053254 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:55.052905 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" event={"ID":"9395f48f-945c-46e5-8c16-6d254bcc9e82","Type":"ContainerDied","Data":"c1fb67760aec00b0504bbaf4f79b487ea2537071203196f8bf09761662e1c237"} Jul 06 23:29:56.177353 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:56.177332 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" Jul 06 23:29:56.352809 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:56.352706 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9395f48f-945c-46e5-8c16-6d254bcc9e82-util\") pod \"9395f48f-945c-46e5-8c16-6d254bcc9e82\" (UID: \"9395f48f-945c-46e5-8c16-6d254bcc9e82\") " Jul 06 23:29:56.352809 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:56.352771 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk27r\" (UniqueName: \"kubernetes.io/projected/9395f48f-945c-46e5-8c16-6d254bcc9e82-kube-api-access-rk27r\") pod \"9395f48f-945c-46e5-8c16-6d254bcc9e82\" (UID: \"9395f48f-945c-46e5-8c16-6d254bcc9e82\") " Jul 06 23:29:56.352952 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:56.352816 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9395f48f-945c-46e5-8c16-6d254bcc9e82-bundle\") pod \"9395f48f-945c-46e5-8c16-6d254bcc9e82\" (UID: \"9395f48f-945c-46e5-8c16-6d254bcc9e82\") " Jul 06 23:29:56.353441 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:56.353412 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9395f48f-945c-46e5-8c16-6d254bcc9e82-bundle" (OuterVolumeSpecName: "bundle") pod "9395f48f-945c-46e5-8c16-6d254bcc9e82" (UID: "9395f48f-945c-46e5-8c16-6d254bcc9e82"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:29:56.355139 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:56.355116 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9395f48f-945c-46e5-8c16-6d254bcc9e82-kube-api-access-rk27r" (OuterVolumeSpecName: "kube-api-access-rk27r") pod "9395f48f-945c-46e5-8c16-6d254bcc9e82" (UID: "9395f48f-945c-46e5-8c16-6d254bcc9e82"). InnerVolumeSpecName "kube-api-access-rk27r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:29:56.357427 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:56.357407 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9395f48f-945c-46e5-8c16-6d254bcc9e82-util" (OuterVolumeSpecName: "util") pod "9395f48f-945c-46e5-8c16-6d254bcc9e82" (UID: "9395f48f-945c-46e5-8c16-6d254bcc9e82"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:29:56.453331 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:56.453311 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk27r\" (UniqueName: \"kubernetes.io/projected/9395f48f-945c-46e5-8c16-6d254bcc9e82-kube-api-access-rk27r\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:29:56.453433 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:56.453333 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9395f48f-945c-46e5-8c16-6d254bcc9e82-bundle\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:29:56.453433 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:56.453348 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9395f48f-945c-46e5-8c16-6d254bcc9e82-util\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:29:57.061878 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:57.061842 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" event={"ID":"9395f48f-945c-46e5-8c16-6d254bcc9e82","Type":"ContainerDied","Data":"e303ffb3e0ad57f7a9f0d14bd55a363bbff359d53566ba74e1a99d1b1a9c3c88"} Jul 06 23:29:57.061878 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:57.061864 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqxh4" Jul 06 23:29:57.061878 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:29:57.061887 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e303ffb3e0ad57f7a9f0d14bd55a363bbff359d53566ba74e1a99d1b1a9c3c88" Jul 06 23:30:00.040533 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.040500 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-49cgw"] Jul 06 23:30:00.040928 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.040742 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9395f48f-945c-46e5-8c16-6d254bcc9e82" containerName="util" Jul 06 23:30:00.040928 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.040755 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9395f48f-945c-46e5-8c16-6d254bcc9e82" containerName="util" Jul 06 23:30:00.040928 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.040763 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9395f48f-945c-46e5-8c16-6d254bcc9e82" containerName="pull" Jul 06 23:30:00.040928 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.040768 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9395f48f-945c-46e5-8c16-6d254bcc9e82" containerName="pull" Jul 06 23:30:00.040928 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.040779 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9395f48f-945c-46e5-8c16-6d254bcc9e82" containerName="extract" Jul 06 23:30:00.040928 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.040785 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9395f48f-945c-46e5-8c16-6d254bcc9e82" containerName="extract" Jul 06 23:30:00.040928 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.040832 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9395f48f-945c-46e5-8c16-6d254bcc9e82" containerName="extract" Jul 06 23:30:00.071206 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.071183 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-49cgw"] Jul 06 23:30:00.071330 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.071271 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-49cgw" Jul 06 23:30:00.074360 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.074341 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:30:00.074548 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.074531 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-zzvwj\"" Jul 06 23:30:00.074907 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.074889 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 06 23:30:00.080645 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.080625 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e487e12-432d-43fc-8ab1-7352be70f9fd-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-49cgw\" (UID: \"0e487e12-432d-43fc-8ab1-7352be70f9fd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-49cgw" Jul 06 23:30:00.080748 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.080663 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj8kx\" (UniqueName: \"kubernetes.io/projected/0e487e12-432d-43fc-8ab1-7352be70f9fd-kube-api-access-fj8kx\") pod \"cert-manager-operator-controller-manager-55c444bf86-49cgw\" (UID: \"0e487e12-432d-43fc-8ab1-7352be70f9fd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-49cgw" Jul 06 23:30:00.181139 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.181113 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e487e12-432d-43fc-8ab1-7352be70f9fd-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-49cgw\" (UID: \"0e487e12-432d-43fc-8ab1-7352be70f9fd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-49cgw" Jul 06 23:30:00.181285 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.181164 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj8kx\" (UniqueName: \"kubernetes.io/projected/0e487e12-432d-43fc-8ab1-7352be70f9fd-kube-api-access-fj8kx\") pod \"cert-manager-operator-controller-manager-55c444bf86-49cgw\" (UID: \"0e487e12-432d-43fc-8ab1-7352be70f9fd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-49cgw" Jul 06 23:30:00.181470 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.181448 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e487e12-432d-43fc-8ab1-7352be70f9fd-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-49cgw\" (UID: \"0e487e12-432d-43fc-8ab1-7352be70f9fd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-49cgw" Jul 06 23:30:00.190419 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.190401 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj8kx\" (UniqueName: \"kubernetes.io/projected/0e487e12-432d-43fc-8ab1-7352be70f9fd-kube-api-access-fj8kx\") pod \"cert-manager-operator-controller-manager-55c444bf86-49cgw\" (UID: \"0e487e12-432d-43fc-8ab1-7352be70f9fd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-49cgw" Jul 06 23:30:00.380014 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.379970 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-49cgw" Jul 06 23:30:00.530987 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.529430 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-49cgw"] Jul 06 23:30:00.535753 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:30:00.535713 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e487e12_432d_43fc_8ab1_7352be70f9fd.slice/crio-b863d31ba153e5ef49c044c910188c53f8b94c55f77a0f7f1e094cf2ec179149 WatchSource:0}: Error finding container b863d31ba153e5ef49c044c910188c53f8b94c55f77a0f7f1e094cf2ec179149: Status 404 returned error can't find the container with id b863d31ba153e5ef49c044c910188c53f8b94c55f77a0f7f1e094cf2ec179149 Jul 06 23:30:00.541256 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:00.541236 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 23:30:01.078201 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:01.078163 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-49cgw" event={"ID":"0e487e12-432d-43fc-8ab1-7352be70f9fd","Type":"ContainerStarted","Data":"b863d31ba153e5ef49c044c910188c53f8b94c55f77a0f7f1e094cf2ec179149"} Jul 06 23:30:03.087201 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:03.087163 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-49cgw" event={"ID":"0e487e12-432d-43fc-8ab1-7352be70f9fd","Type":"ContainerStarted","Data":"e64098bfe1d63a60d9febfa546b6c2adb296e91248d380eb75b742ce7cb386dd"} Jul 06 23:30:03.110435 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:03.110379 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-49cgw" podStartSLOduration=1.262442306 podStartE2EDuration="3.110358557s" totalImagesPullingTime="1.847916251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:30:03.10792524 +0000 UTC m=+313.821979815" watchObservedRunningTime="2026-07-06 23:30:03.110358557 +0000 UTC m=+313.824413128" Jul 06 23:30:04.259783 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:04.259746 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7"] Jul 06 23:30:04.282833 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:04.282807 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7"] Jul 06 23:30:04.282983 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:04.282907 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7" Jul 06 23:30:04.286019 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:04.285997 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 23:30:04.286148 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:04.286018 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cf5fj\"" Jul 06 23:30:04.286752 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:04.286712 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 23:30:04.308809 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:04.308783 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b94f8c0-b74c-4942-96e1-ee95210db1f4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7\" (UID: \"3b94f8c0-b74c-4942-96e1-ee95210db1f4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7" Jul 06 23:30:04.308911 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:04.308824 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdcv4\" (UniqueName: \"kubernetes.io/projected/3b94f8c0-b74c-4942-96e1-ee95210db1f4-kube-api-access-rdcv4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7\" (UID: \"3b94f8c0-b74c-4942-96e1-ee95210db1f4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7" Jul 06 23:30:04.308911 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:04.308879 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b94f8c0-b74c-4942-96e1-ee95210db1f4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7\" (UID: \"3b94f8c0-b74c-4942-96e1-ee95210db1f4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7" Jul 06 23:30:04.409490 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:04.409462 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdcv4\" (UniqueName: \"kubernetes.io/projected/3b94f8c0-b74c-4942-96e1-ee95210db1f4-kube-api-access-rdcv4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7\" (UID: \"3b94f8c0-b74c-4942-96e1-ee95210db1f4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7" Jul 06 23:30:04.409647 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:04.409507 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b94f8c0-b74c-4942-96e1-ee95210db1f4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7\" (UID: \"3b94f8c0-b74c-4942-96e1-ee95210db1f4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7" Jul 06 23:30:04.409647 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:04.409549 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b94f8c0-b74c-4942-96e1-ee95210db1f4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7\" (UID: \"3b94f8c0-b74c-4942-96e1-ee95210db1f4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7" Jul 06 23:30:04.409895 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:04.409879 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b94f8c0-b74c-4942-96e1-ee95210db1f4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7\" (UID: \"3b94f8c0-b74c-4942-96e1-ee95210db1f4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7" Jul 06 23:30:04.409939 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:04.409914 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b94f8c0-b74c-4942-96e1-ee95210db1f4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7\" (UID: \"3b94f8c0-b74c-4942-96e1-ee95210db1f4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7" Jul 06 23:30:04.436700 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:04.436673 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdcv4\" (UniqueName: \"kubernetes.io/projected/3b94f8c0-b74c-4942-96e1-ee95210db1f4-kube-api-access-rdcv4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7\" (UID: \"3b94f8c0-b74c-4942-96e1-ee95210db1f4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7" Jul 06 23:30:04.592760 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:04.592656 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7" Jul 06 23:30:04.732588 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:04.732557 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7"] Jul 06 23:30:04.737123 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:30:04.737099 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b94f8c0_b74c_4942_96e1_ee95210db1f4.slice/crio-317a1259f62e37e7798bd0c8798a71b098802315dd558971fe9c6db04ba2dc87 WatchSource:0}: Error finding container 317a1259f62e37e7798bd0c8798a71b098802315dd558971fe9c6db04ba2dc87: Status 404 returned error can't find the container with id 317a1259f62e37e7798bd0c8798a71b098802315dd558971fe9c6db04ba2dc87 Jul 06 23:30:05.096276 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:05.096244 2571 generic.go:356] "Generic (PLEG): container finished" podID="3b94f8c0-b74c-4942-96e1-ee95210db1f4" containerID="bb91513b8b5325777865af4706c9bd7db818cebb4d4807f750890f17557be838" exitCode=0 Jul 06 23:30:05.096447 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:05.096332 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7" event={"ID":"3b94f8c0-b74c-4942-96e1-ee95210db1f4","Type":"ContainerDied","Data":"bb91513b8b5325777865af4706c9bd7db818cebb4d4807f750890f17557be838"} Jul 06 23:30:05.096447 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:05.096385 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7" event={"ID":"3b94f8c0-b74c-4942-96e1-ee95210db1f4","Type":"ContainerStarted","Data":"317a1259f62e37e7798bd0c8798a71b098802315dd558971fe9c6db04ba2dc87"} Jul 06 23:30:08.109536 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:08.109498 2571 generic.go:356] "Generic (PLEG): container finished" podID="3b94f8c0-b74c-4942-96e1-ee95210db1f4" containerID="b50370bf17d52e716b6a70345f37adea7765b91ff94611c37997b528af7fd247" exitCode=0 Jul 06 23:30:08.109931 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:08.109557 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7" event={"ID":"3b94f8c0-b74c-4942-96e1-ee95210db1f4","Type":"ContainerDied","Data":"b50370bf17d52e716b6a70345f37adea7765b91ff94611c37997b528af7fd247"} Jul 06 23:30:09.115180 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:09.115147 2571 generic.go:356] "Generic (PLEG): container finished" podID="3b94f8c0-b74c-4942-96e1-ee95210db1f4" containerID="28dff15b4ca4e5b6cd7b3270d50d35efe95a7e79eb6c4fea235a5e6bf235dd99" exitCode=0 Jul 06 23:30:09.115583 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:09.115224 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7" event={"ID":"3b94f8c0-b74c-4942-96e1-ee95210db1f4","Type":"ContainerDied","Data":"28dff15b4ca4e5b6cd7b3270d50d35efe95a7e79eb6c4fea235a5e6bf235dd99"} Jul 06 23:30:10.242308 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:10.242287 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7" Jul 06 23:30:10.248110 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:10.248089 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b94f8c0-b74c-4942-96e1-ee95210db1f4-util\") pod \"3b94f8c0-b74c-4942-96e1-ee95210db1f4\" (UID: \"3b94f8c0-b74c-4942-96e1-ee95210db1f4\") " Jul 06 23:30:10.248221 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:10.248129 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdcv4\" (UniqueName: \"kubernetes.io/projected/3b94f8c0-b74c-4942-96e1-ee95210db1f4-kube-api-access-rdcv4\") pod \"3b94f8c0-b74c-4942-96e1-ee95210db1f4\" (UID: \"3b94f8c0-b74c-4942-96e1-ee95210db1f4\") " Jul 06 23:30:10.248221 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:10.248200 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b94f8c0-b74c-4942-96e1-ee95210db1f4-bundle\") pod \"3b94f8c0-b74c-4942-96e1-ee95210db1f4\" (UID: \"3b94f8c0-b74c-4942-96e1-ee95210db1f4\") " Jul 06 23:30:10.248578 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:10.248555 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b94f8c0-b74c-4942-96e1-ee95210db1f4-bundle" (OuterVolumeSpecName: "bundle") pod "3b94f8c0-b74c-4942-96e1-ee95210db1f4" (UID: "3b94f8c0-b74c-4942-96e1-ee95210db1f4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:30:10.250301 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:10.250277 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b94f8c0-b74c-4942-96e1-ee95210db1f4-kube-api-access-rdcv4" (OuterVolumeSpecName: "kube-api-access-rdcv4") pod "3b94f8c0-b74c-4942-96e1-ee95210db1f4" (UID: "3b94f8c0-b74c-4942-96e1-ee95210db1f4"). InnerVolumeSpecName "kube-api-access-rdcv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:30:10.252668 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:10.252633 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b94f8c0-b74c-4942-96e1-ee95210db1f4-util" (OuterVolumeSpecName: "util") pod "3b94f8c0-b74c-4942-96e1-ee95210db1f4" (UID: "3b94f8c0-b74c-4942-96e1-ee95210db1f4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:30:10.348888 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:10.348852 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b94f8c0-b74c-4942-96e1-ee95210db1f4-bundle\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:30:10.348888 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:10.348880 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b94f8c0-b74c-4942-96e1-ee95210db1f4-util\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:30:10.348888 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:10.348893 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdcv4\" (UniqueName: \"kubernetes.io/projected/3b94f8c0-b74c-4942-96e1-ee95210db1f4-kube-api-access-rdcv4\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:30:11.124429 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:11.124389 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7" event={"ID":"3b94f8c0-b74c-4942-96e1-ee95210db1f4","Type":"ContainerDied","Data":"317a1259f62e37e7798bd0c8798a71b098802315dd558971fe9c6db04ba2dc87"} Jul 06 23:30:11.124611 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:11.124434 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqnrp7" Jul 06 23:30:11.124611 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:11.124438 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="317a1259f62e37e7798bd0c8798a71b098802315dd558971fe9c6db04ba2dc87" Jul 06 23:30:13.952602 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:13.952568 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pgm8p"] Jul 06 23:30:13.953002 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:13.952847 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b94f8c0-b74c-4942-96e1-ee95210db1f4" containerName="util" Jul 06 23:30:13.953002 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:13.952859 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b94f8c0-b74c-4942-96e1-ee95210db1f4" containerName="util" Jul 06 23:30:13.953002 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:13.952866 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b94f8c0-b74c-4942-96e1-ee95210db1f4" containerName="pull" Jul 06 23:30:13.953002 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:13.952872 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b94f8c0-b74c-4942-96e1-ee95210db1f4" containerName="pull" Jul 06 23:30:13.953002 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:13.952880 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b94f8c0-b74c-4942-96e1-ee95210db1f4" containerName="extract" Jul 06 23:30:13.953002 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:13.952885 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b94f8c0-b74c-4942-96e1-ee95210db1f4" containerName="extract" Jul 06 23:30:13.953002 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:13.952925 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b94f8c0-b74c-4942-96e1-ee95210db1f4" containerName="extract" Jul 06 23:30:13.954664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:13.954649 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pgm8p" Jul 06 23:30:13.956870 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:13.956846 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 23:30:13.956985 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:13.956904 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 23:30:13.957906 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:13.957882 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-j95dn\"" Jul 06 23:30:13.966250 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:13.966227 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pgm8p"] Jul 06 23:30:13.977417 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:13.977394 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09791643-ed93-4a04-a085-718254e6b2c5-bound-sa-token\") pod \"cert-manager-55b4d954d5-pgm8p\" (UID: \"09791643-ed93-4a04-a085-718254e6b2c5\") " pod="cert-manager/cert-manager-55b4d954d5-pgm8p" Jul 06 23:30:13.977536 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:13.977432 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m42k2\" (UniqueName: \"kubernetes.io/projected/09791643-ed93-4a04-a085-718254e6b2c5-kube-api-access-m42k2\") pod \"cert-manager-55b4d954d5-pgm8p\" (UID: \"09791643-ed93-4a04-a085-718254e6b2c5\") " pod="cert-manager/cert-manager-55b4d954d5-pgm8p" Jul 06 23:30:14.078139 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:14.078098 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09791643-ed93-4a04-a085-718254e6b2c5-bound-sa-token\") pod \"cert-manager-55b4d954d5-pgm8p\" (UID: \"09791643-ed93-4a04-a085-718254e6b2c5\") " pod="cert-manager/cert-manager-55b4d954d5-pgm8p" Jul 06 23:30:14.078336 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:14.078154 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m42k2\" (UniqueName: \"kubernetes.io/projected/09791643-ed93-4a04-a085-718254e6b2c5-kube-api-access-m42k2\") pod \"cert-manager-55b4d954d5-pgm8p\" (UID: \"09791643-ed93-4a04-a085-718254e6b2c5\") " pod="cert-manager/cert-manager-55b4d954d5-pgm8p" Jul 06 23:30:14.086859 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:14.086821 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09791643-ed93-4a04-a085-718254e6b2c5-bound-sa-token\") pod \"cert-manager-55b4d954d5-pgm8p\" (UID: \"09791643-ed93-4a04-a085-718254e6b2c5\") " pod="cert-manager/cert-manager-55b4d954d5-pgm8p" Jul 06 23:30:14.087128 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:14.087108 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m42k2\" (UniqueName: \"kubernetes.io/projected/09791643-ed93-4a04-a085-718254e6b2c5-kube-api-access-m42k2\") pod \"cert-manager-55b4d954d5-pgm8p\" (UID: \"09791643-ed93-4a04-a085-718254e6b2c5\") " pod="cert-manager/cert-manager-55b4d954d5-pgm8p" Jul 06 23:30:14.263525 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:14.263436 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pgm8p" Jul 06 23:30:14.390963 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:14.390939 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pgm8p"] Jul 06 23:30:14.392430 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:30:14.392401 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09791643_ed93_4a04_a085_718254e6b2c5.slice/crio-df4b382dfac8b4f5c46f8886b34bd32535f51a32cf107967d633149a3598dcd1 WatchSource:0}: Error finding container df4b382dfac8b4f5c46f8886b34bd32535f51a32cf107967d633149a3598dcd1: Status 404 returned error can't find the container with id df4b382dfac8b4f5c46f8886b34bd32535f51a32cf107967d633149a3598dcd1 Jul 06 23:30:15.142502 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:15.142464 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pgm8p" event={"ID":"09791643-ed93-4a04-a085-718254e6b2c5","Type":"ContainerStarted","Data":"df4b382dfac8b4f5c46f8886b34bd32535f51a32cf107967d633149a3598dcd1"} Jul 06 23:30:17.152420 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:17.152339 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pgm8p" event={"ID":"09791643-ed93-4a04-a085-718254e6b2c5","Type":"ContainerStarted","Data":"3c9e4c7ee9a222a051976fd86cea8acdd9ed5915bcc783b340f45e85f5ddce51"} Jul 06 23:30:17.171981 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:17.171941 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-pgm8p" podStartSLOduration=1.82925139 podStartE2EDuration="4.171925729s" totalImagesPullingTime="2.342674339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:30:17.171496255 +0000 UTC m=+327.885550823" watchObservedRunningTime="2026-07-06 23:30:17.171925729 +0000 UTC m=+327.885980297" Jul 06 23:30:23.346001 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:23.345972 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk"] Jul 06 23:30:23.348123 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:23.348105 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk" Jul 06 23:30:23.350287 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:23.350254 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 23:30:23.350380 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:23.350358 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 23:30:23.351182 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:23.351165 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cf5fj\"" Jul 06 23:30:23.357912 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:23.357889 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk"] Jul 06 23:30:23.442894 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:23.442865 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6aec2d17-eff0-4317-b7e7-c204a12f524a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk\" (UID: \"6aec2d17-eff0-4317-b7e7-c204a12f524a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk" Jul 06 23:30:23.443013 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:23.442905 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6aec2d17-eff0-4317-b7e7-c204a12f524a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk\" (UID: \"6aec2d17-eff0-4317-b7e7-c204a12f524a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk" Jul 06 23:30:23.443013 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:23.442999 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj2m8\" (UniqueName: \"kubernetes.io/projected/6aec2d17-eff0-4317-b7e7-c204a12f524a-kube-api-access-dj2m8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk\" (UID: \"6aec2d17-eff0-4317-b7e7-c204a12f524a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk" Jul 06 23:30:23.543831 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:23.543806 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6aec2d17-eff0-4317-b7e7-c204a12f524a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk\" (UID: \"6aec2d17-eff0-4317-b7e7-c204a12f524a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk" Jul 06 23:30:23.543952 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:23.543851 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6aec2d17-eff0-4317-b7e7-c204a12f524a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk\" (UID: \"6aec2d17-eff0-4317-b7e7-c204a12f524a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk" Jul 06 23:30:23.543952 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:23.543892 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj2m8\" (UniqueName: \"kubernetes.io/projected/6aec2d17-eff0-4317-b7e7-c204a12f524a-kube-api-access-dj2m8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk\" (UID: \"6aec2d17-eff0-4317-b7e7-c204a12f524a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk" Jul 06 23:30:23.544243 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:23.544219 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6aec2d17-eff0-4317-b7e7-c204a12f524a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk\" (UID: \"6aec2d17-eff0-4317-b7e7-c204a12f524a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk" Jul 06 23:30:23.544313 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:23.544231 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6aec2d17-eff0-4317-b7e7-c204a12f524a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk\" (UID: \"6aec2d17-eff0-4317-b7e7-c204a12f524a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk" Jul 06 23:30:23.552317 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:23.552296 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj2m8\" (UniqueName: \"kubernetes.io/projected/6aec2d17-eff0-4317-b7e7-c204a12f524a-kube-api-access-dj2m8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk\" (UID: \"6aec2d17-eff0-4317-b7e7-c204a12f524a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk" Jul 06 23:30:23.657731 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:23.657650 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk" Jul 06 23:30:23.791224 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:23.791201 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk"] Jul 06 23:30:24.181331 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:24.181294 2571 generic.go:356] "Generic (PLEG): container finished" podID="6aec2d17-eff0-4317-b7e7-c204a12f524a" containerID="6e1f4382543a6e3ecb87b7c741e1419620898a7d2ec99f16862eab5a32aac719" exitCode=0 Jul 06 23:30:24.181516 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:24.181379 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk" event={"ID":"6aec2d17-eff0-4317-b7e7-c204a12f524a","Type":"ContainerDied","Data":"6e1f4382543a6e3ecb87b7c741e1419620898a7d2ec99f16862eab5a32aac719"} Jul 06 23:30:24.181516 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:24.181440 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk" event={"ID":"6aec2d17-eff0-4317-b7e7-c204a12f524a","Type":"ContainerStarted","Data":"3254fa185506c4bbcceb1de183c0fede53255b1c957119f06e9405f0cfcd6be7"} Jul 06 23:30:25.187647 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:25.187613 2571 generic.go:356] "Generic (PLEG): container finished" podID="6aec2d17-eff0-4317-b7e7-c204a12f524a" containerID="d7361367acb2a10d031b20e7344b6e3a1a740d1c4cdaa7ecc344c4f9e6ad8ad6" exitCode=0 Jul 06 23:30:25.188046 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:25.187707 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk" event={"ID":"6aec2d17-eff0-4317-b7e7-c204a12f524a","Type":"ContainerDied","Data":"d7361367acb2a10d031b20e7344b6e3a1a740d1c4cdaa7ecc344c4f9e6ad8ad6"} Jul 06 23:30:26.193907 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:26.193871 2571 generic.go:356] "Generic (PLEG): container finished" podID="6aec2d17-eff0-4317-b7e7-c204a12f524a" containerID="dff12a6ee142d1e797ab4867c849eb789ad7e47b57c4c1ef4db28c463a535955" exitCode=0 Jul 06 23:30:26.194311 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:26.193958 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk" event={"ID":"6aec2d17-eff0-4317-b7e7-c204a12f524a","Type":"ContainerDied","Data":"dff12a6ee142d1e797ab4867c849eb789ad7e47b57c4c1ef4db28c463a535955"} Jul 06 23:30:27.321030 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:27.321006 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk" Jul 06 23:30:27.373372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:27.373343 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6aec2d17-eff0-4317-b7e7-c204a12f524a-util\") pod \"6aec2d17-eff0-4317-b7e7-c204a12f524a\" (UID: \"6aec2d17-eff0-4317-b7e7-c204a12f524a\") " Jul 06 23:30:27.373474 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:27.373406 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj2m8\" (UniqueName: \"kubernetes.io/projected/6aec2d17-eff0-4317-b7e7-c204a12f524a-kube-api-access-dj2m8\") pod \"6aec2d17-eff0-4317-b7e7-c204a12f524a\" (UID: \"6aec2d17-eff0-4317-b7e7-c204a12f524a\") " Jul 06 23:30:27.373474 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:27.373465 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6aec2d17-eff0-4317-b7e7-c204a12f524a-bundle\") pod \"6aec2d17-eff0-4317-b7e7-c204a12f524a\" (UID: \"6aec2d17-eff0-4317-b7e7-c204a12f524a\") " Jul 06 23:30:27.374146 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:27.374116 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aec2d17-eff0-4317-b7e7-c204a12f524a-bundle" (OuterVolumeSpecName: "bundle") pod "6aec2d17-eff0-4317-b7e7-c204a12f524a" (UID: "6aec2d17-eff0-4317-b7e7-c204a12f524a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:30:27.375535 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:27.375514 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aec2d17-eff0-4317-b7e7-c204a12f524a-kube-api-access-dj2m8" (OuterVolumeSpecName: "kube-api-access-dj2m8") pod "6aec2d17-eff0-4317-b7e7-c204a12f524a" (UID: "6aec2d17-eff0-4317-b7e7-c204a12f524a"). InnerVolumeSpecName "kube-api-access-dj2m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:30:27.378828 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:27.378704 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aec2d17-eff0-4317-b7e7-c204a12f524a-util" (OuterVolumeSpecName: "util") pod "6aec2d17-eff0-4317-b7e7-c204a12f524a" (UID: "6aec2d17-eff0-4317-b7e7-c204a12f524a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:30:27.474135 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:27.474050 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj2m8\" (UniqueName: \"kubernetes.io/projected/6aec2d17-eff0-4317-b7e7-c204a12f524a-kube-api-access-dj2m8\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:30:27.474135 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:27.474081 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6aec2d17-eff0-4317-b7e7-c204a12f524a-bundle\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:30:27.474135 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:27.474090 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6aec2d17-eff0-4317-b7e7-c204a12f524a-util\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:30:28.203431 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:28.203391 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk" event={"ID":"6aec2d17-eff0-4317-b7e7-c204a12f524a","Type":"ContainerDied","Data":"3254fa185506c4bbcceb1de183c0fede53255b1c957119f06e9405f0cfcd6be7"} Jul 06 23:30:28.203584 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:28.203435 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpdxrk" Jul 06 23:30:28.203584 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:28.203439 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3254fa185506c4bbcceb1de183c0fede53255b1c957119f06e9405f0cfcd6be7" Jul 06 23:30:35.998258 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:35.998225 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2"] Jul 06 23:30:35.998680 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:35.998470 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aec2d17-eff0-4317-b7e7-c204a12f524a" containerName="util" Jul 06 23:30:35.998680 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:35.998480 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aec2d17-eff0-4317-b7e7-c204a12f524a" containerName="util" Jul 06 23:30:35.998680 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:35.998487 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aec2d17-eff0-4317-b7e7-c204a12f524a" containerName="extract" Jul 06 23:30:35.998680 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:35.998492 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aec2d17-eff0-4317-b7e7-c204a12f524a" containerName="extract" Jul 06 23:30:35.998680 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:35.998505 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aec2d17-eff0-4317-b7e7-c204a12f524a" containerName="pull" Jul 06 23:30:35.998680 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:35.998510 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aec2d17-eff0-4317-b7e7-c204a12f524a" containerName="pull" Jul 06 23:30:35.998680 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:35.998556 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aec2d17-eff0-4317-b7e7-c204a12f524a" containerName="extract" Jul 06 23:30:36.001269 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.001249 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" Jul 06 23:30:36.004836 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.004812 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 23:30:36.004951 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.004894 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 23:30:36.005022 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.004954 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 23:30:36.005022 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.004978 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 23:30:36.005133 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.005021 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:30:36.005185 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.005137 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-gpbg9\"" Jul 06 23:30:36.022218 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.022192 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2"] Jul 06 23:30:36.037990 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.037962 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd-metrics-cert\") pod \"lws-controller-manager-7ff7c895fb-7tbp2\" (UID: \"7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd\") " pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" Jul 06 23:30:36.038105 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.037999 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mmwb\" (UniqueName: \"kubernetes.io/projected/7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd-kube-api-access-6mmwb\") pod \"lws-controller-manager-7ff7c895fb-7tbp2\" (UID: \"7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd\") " pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" Jul 06 23:30:36.038105 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.038046 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd-cert\") pod \"lws-controller-manager-7ff7c895fb-7tbp2\" (UID: \"7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd\") " pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" Jul 06 23:30:36.038105 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.038068 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd-manager-config\") pod \"lws-controller-manager-7ff7c895fb-7tbp2\" (UID: \"7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd\") " pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" Jul 06 23:30:36.138612 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.138579 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mmwb\" (UniqueName: \"kubernetes.io/projected/7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd-kube-api-access-6mmwb\") pod \"lws-controller-manager-7ff7c895fb-7tbp2\" (UID: \"7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd\") " pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" Jul 06 23:30:36.138612 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.138617 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd-cert\") pod \"lws-controller-manager-7ff7c895fb-7tbp2\" (UID: \"7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd\") " pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" Jul 06 23:30:36.138844 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.138646 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd-manager-config\") pod \"lws-controller-manager-7ff7c895fb-7tbp2\" (UID: \"7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd\") " pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" Jul 06 23:30:36.138844 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.138703 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd-metrics-cert\") pod \"lws-controller-manager-7ff7c895fb-7tbp2\" (UID: \"7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd\") " pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" Jul 06 23:30:36.139391 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.139368 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd-manager-config\") pod \"lws-controller-manager-7ff7c895fb-7tbp2\" (UID: \"7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd\") " pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" Jul 06 23:30:36.141401 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.141371 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd-cert\") pod \"lws-controller-manager-7ff7c895fb-7tbp2\" (UID: \"7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd\") " pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" Jul 06 23:30:36.141503 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.141371 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd-metrics-cert\") pod \"lws-controller-manager-7ff7c895fb-7tbp2\" (UID: \"7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd\") " pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" Jul 06 23:30:36.149076 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.149053 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mmwb\" (UniqueName: \"kubernetes.io/projected/7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd-kube-api-access-6mmwb\") pod \"lws-controller-manager-7ff7c895fb-7tbp2\" (UID: \"7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd\") " pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" Jul 06 23:30:36.310236 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.310155 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" Jul 06 23:30:36.461844 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:36.461822 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2"] Jul 06 23:30:36.464135 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:30:36.464105 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5b93ea_f4b1_4b73_a362_98a6ac4cf6dd.slice/crio-3693c607cb56cba3d092859b9bb62e2843898b299b507c8c69f76897ff5e706b WatchSource:0}: Error finding container 3693c607cb56cba3d092859b9bb62e2843898b299b507c8c69f76897ff5e706b: Status 404 returned error can't find the container with id 3693c607cb56cba3d092859b9bb62e2843898b299b507c8c69f76897ff5e706b Jul 06 23:30:37.242240 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.242203 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" event={"ID":"7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd","Type":"ContainerStarted","Data":"3693c607cb56cba3d092859b9bb62e2843898b299b507c8c69f76897ff5e706b"} Jul 06 23:30:37.335108 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.335071 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k"] Jul 06 23:30:37.339107 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.339084 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:30:37.342399 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.342362 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 23:30:37.342599 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.342585 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 23:30:37.342743 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.342703 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 23:30:37.346809 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.346773 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8sd44\"" Jul 06 23:30:37.349948 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.349924 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 23:30:37.384152 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.384115 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k"] Jul 06 23:30:37.448004 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.447970 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf5nc\" (UniqueName: \"kubernetes.io/projected/2a628d66-50b8-4d07-a2c9-3143e8a02ceb-kube-api-access-nf5nc\") pod \"opendatahub-operator-controller-manager-d555ff84d-wq97k\" (UID: \"2a628d66-50b8-4d07-a2c9-3143e8a02ceb\") " pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:30:37.448172 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.448021 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2a628d66-50b8-4d07-a2c9-3143e8a02ceb-webhook-cert\") pod \"opendatahub-operator-controller-manager-d555ff84d-wq97k\" (UID: \"2a628d66-50b8-4d07-a2c9-3143e8a02ceb\") " pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:30:37.448172 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.448099 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2a628d66-50b8-4d07-a2c9-3143e8a02ceb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d555ff84d-wq97k\" (UID: \"2a628d66-50b8-4d07-a2c9-3143e8a02ceb\") " pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:30:37.448279 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.448173 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a628d66-50b8-4d07-a2c9-3143e8a02ceb-tmp\") pod \"opendatahub-operator-controller-manager-d555ff84d-wq97k\" (UID: \"2a628d66-50b8-4d07-a2c9-3143e8a02ceb\") " pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:30:37.448279 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.448207 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2a628d66-50b8-4d07-a2c9-3143e8a02ceb-manifests\") pod \"opendatahub-operator-controller-manager-d555ff84d-wq97k\" (UID: \"2a628d66-50b8-4d07-a2c9-3143e8a02ceb\") " pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:30:37.549470 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.549382 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2a628d66-50b8-4d07-a2c9-3143e8a02ceb-manifests\") pod \"opendatahub-operator-controller-manager-d555ff84d-wq97k\" (UID: \"2a628d66-50b8-4d07-a2c9-3143e8a02ceb\") " pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:30:37.549470 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.549439 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf5nc\" (UniqueName: \"kubernetes.io/projected/2a628d66-50b8-4d07-a2c9-3143e8a02ceb-kube-api-access-nf5nc\") pod \"opendatahub-operator-controller-manager-d555ff84d-wq97k\" (UID: \"2a628d66-50b8-4d07-a2c9-3143e8a02ceb\") " pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:30:37.549470 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.549464 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2a628d66-50b8-4d07-a2c9-3143e8a02ceb-webhook-cert\") pod \"opendatahub-operator-controller-manager-d555ff84d-wq97k\" (UID: \"2a628d66-50b8-4d07-a2c9-3143e8a02ceb\") " pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:30:37.549789 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.549495 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2a628d66-50b8-4d07-a2c9-3143e8a02ceb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d555ff84d-wq97k\" (UID: \"2a628d66-50b8-4d07-a2c9-3143e8a02ceb\") " pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:30:37.549789 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.549565 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a628d66-50b8-4d07-a2c9-3143e8a02ceb-tmp\") pod \"opendatahub-operator-controller-manager-d555ff84d-wq97k\" (UID: \"2a628d66-50b8-4d07-a2c9-3143e8a02ceb\") " pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:30:37.549893 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.549868 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2a628d66-50b8-4d07-a2c9-3143e8a02ceb-manifests\") pod \"opendatahub-operator-controller-manager-d555ff84d-wq97k\" (UID: \"2a628d66-50b8-4d07-a2c9-3143e8a02ceb\") " pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:30:37.552309 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.552287 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a628d66-50b8-4d07-a2c9-3143e8a02ceb-tmp\") pod \"opendatahub-operator-controller-manager-d555ff84d-wq97k\" (UID: \"2a628d66-50b8-4d07-a2c9-3143e8a02ceb\") " pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:30:37.552483 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.552464 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2a628d66-50b8-4d07-a2c9-3143e8a02ceb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d555ff84d-wq97k\" (UID: \"2a628d66-50b8-4d07-a2c9-3143e8a02ceb\") " pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:30:37.552944 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.552925 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2a628d66-50b8-4d07-a2c9-3143e8a02ceb-webhook-cert\") pod \"opendatahub-operator-controller-manager-d555ff84d-wq97k\" (UID: \"2a628d66-50b8-4d07-a2c9-3143e8a02ceb\") " pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:30:37.562274 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.562249 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf5nc\" (UniqueName: \"kubernetes.io/projected/2a628d66-50b8-4d07-a2c9-3143e8a02ceb-kube-api-access-nf5nc\") pod \"opendatahub-operator-controller-manager-d555ff84d-wq97k\" (UID: \"2a628d66-50b8-4d07-a2c9-3143e8a02ceb\") " pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:30:37.650414 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.650379 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:30:37.811138 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:37.811045 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k"] Jul 06 23:30:37.815954 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:30:37.815926 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a628d66_50b8_4d07_a2c9_3143e8a02ceb.slice/crio-3eb02b1260b0ddaa1424aef92bd2099b58f0301efcab710f39be12785d0d7e58 WatchSource:0}: Error finding container 3eb02b1260b0ddaa1424aef92bd2099b58f0301efcab710f39be12785d0d7e58: Status 404 returned error can't find the container with id 3eb02b1260b0ddaa1424aef92bd2099b58f0301efcab710f39be12785d0d7e58 Jul 06 23:30:38.247441 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:38.247401 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" event={"ID":"2a628d66-50b8-4d07-a2c9-3143e8a02ceb","Type":"ContainerStarted","Data":"3eb02b1260b0ddaa1424aef92bd2099b58f0301efcab710f39be12785d0d7e58"} Jul 06 23:30:38.248838 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:38.248816 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" event={"ID":"7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd","Type":"ContainerStarted","Data":"c63fbb259e3997eee53ee3ebbf13bebed2ce65d16eb4673d7d37f1721f178106"} Jul 06 23:30:38.248970 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:38.248954 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" Jul 06 23:30:38.274534 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:38.274485 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" podStartSLOduration=2.019084445 podStartE2EDuration="3.274467685s" totalImagesPullingTime="1.25538324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:30:38.272501506 +0000 UTC m=+348.986556078" watchObservedRunningTime="2026-07-06 23:30:38.274467685 +0000 UTC m=+348.988522255" Jul 06 23:30:41.264460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:41.264426 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" event={"ID":"2a628d66-50b8-4d07-a2c9-3143e8a02ceb","Type":"ContainerStarted","Data":"22de945fa3b874fa985667fa1b3c35b5a25066222f523961876b864f9c7b7bc1"} Jul 06 23:30:42.269694 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:42.269662 2571 generic.go:356] "Generic (PLEG): container finished" podID="2a628d66-50b8-4d07-a2c9-3143e8a02ceb" containerID="22de945fa3b874fa985667fa1b3c35b5a25066222f523961876b864f9c7b7bc1" exitCode=0 Jul 06 23:30:42.270098 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:42.269758 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" event={"ID":"2a628d66-50b8-4d07-a2c9-3143e8a02ceb","Type":"ContainerDied","Data":"22de945fa3b874fa985667fa1b3c35b5a25066222f523961876b864f9c7b7bc1"} Jul 06 23:30:43.280817 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:43.280773 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" event={"ID":"2a628d66-50b8-4d07-a2c9-3143e8a02ceb","Type":"ContainerStarted","Data":"4869f156cf8f063eb5f54fd1b35fc0e03ce225eeeb63d46592664c76bca1ade5"} Jul 06 23:30:43.281207 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:43.280890 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:30:43.307087 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:43.307027 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" podStartSLOduration=2.143314844 podStartE2EDuration="6.307008825s" totalImagesPullingTime="3.140225634s" totalInitContainerRuntime="1.023468347s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:30:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:30:43.305364761 +0000 UTC m=+354.019419337" watchObservedRunningTime="2026-07-06 23:30:43.307008825 +0000 UTC m=+354.021063395" Jul 06 23:30:49.256702 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:49.256674 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7ff7c895fb-7tbp2" Jul 06 23:30:54.287213 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:30:54.287185 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d555ff84d-wq97k" Jul 06 23:31:15.198904 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.198869 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn"] Jul 06 23:31:15.254178 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.254145 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn"] Jul 06 23:31:15.254329 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.254245 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.256707 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.256678 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 23:31:15.256853 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.256776 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-ks6nw\"" Jul 06 23:31:15.256949 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.256932 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 23:31:15.257065 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.257024 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 23:31:15.352387 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.352354 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89db98c1-0628-4d66-a459-c80ad509c607-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.352554 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.352397 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89db98c1-0628-4d66-a459-c80ad509c607-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.352554 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.352416 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89db98c1-0628-4d66-a459-c80ad509c607-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.352554 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.352446 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89db98c1-0628-4d66-a459-c80ad509c607-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.352554 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.352513 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89db98c1-0628-4d66-a459-c80ad509c607-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.352685 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.352555 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89db98c1-0628-4d66-a459-c80ad509c607-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.352685 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.352571 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89db98c1-0628-4d66-a459-c80ad509c607-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.352685 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.352599 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89db98c1-0628-4d66-a459-c80ad509c607-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.352685 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.352650 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bqgx\" (UniqueName: \"kubernetes.io/projected/89db98c1-0628-4d66-a459-c80ad509c607-kube-api-access-7bqgx\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.454046 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.453968 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89db98c1-0628-4d66-a459-c80ad509c607-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.454046 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.454003 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89db98c1-0628-4d66-a459-c80ad509c607-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.454046 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.454027 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89db98c1-0628-4d66-a459-c80ad509c607-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.454269 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.454066 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bqgx\" (UniqueName: \"kubernetes.io/projected/89db98c1-0628-4d66-a459-c80ad509c607-kube-api-access-7bqgx\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.454313 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.454257 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89db98c1-0628-4d66-a459-c80ad509c607-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.454313 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.454304 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89db98c1-0628-4d66-a459-c80ad509c607-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.454417 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.454328 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89db98c1-0628-4d66-a459-c80ad509c607-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.454417 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.454330 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89db98c1-0628-4d66-a459-c80ad509c607-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.454417 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.454357 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89db98c1-0628-4d66-a459-c80ad509c607-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.454417 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.454384 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89db98c1-0628-4d66-a459-c80ad509c607-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.454598 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.454583 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89db98c1-0628-4d66-a459-c80ad509c607-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.454656 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.454636 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89db98c1-0628-4d66-a459-c80ad509c607-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.454781 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.454758 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89db98c1-0628-4d66-a459-c80ad509c607-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.454891 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.454788 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89db98c1-0628-4d66-a459-c80ad509c607-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.456929 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.456900 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89db98c1-0628-4d66-a459-c80ad509c607-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.457007 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.456958 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89db98c1-0628-4d66-a459-c80ad509c607-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.462700 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.462676 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89db98c1-0628-4d66-a459-c80ad509c607-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.462917 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.462893 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bqgx\" (UniqueName: \"kubernetes.io/projected/89db98c1-0628-4d66-a459-c80ad509c607-kube-api-access-7bqgx\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48kwjn\" (UID: \"89db98c1-0628-4d66-a459-c80ad509c607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.564606 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.564573 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:15.723488 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:15.723412 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn"] Jul 06 23:31:15.726747 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:31:15.726687 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89db98c1_0628_4d66_a459_c80ad509c607.slice/crio-e9f0103d5649bed7e228fa508321978d226f18cddebc916597aeb7656d21940f WatchSource:0}: Error finding container e9f0103d5649bed7e228fa508321978d226f18cddebc916597aeb7656d21940f: Status 404 returned error can't find the container with id e9f0103d5649bed7e228fa508321978d226f18cddebc916597aeb7656d21940f Jul 06 23:31:16.426179 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:16.426103 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" event={"ID":"89db98c1-0628-4d66-a459-c80ad509c607","Type":"ContainerStarted","Data":"e9f0103d5649bed7e228fa508321978d226f18cddebc916597aeb7656d21940f"} Jul 06 23:31:18.358868 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:18.358826 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:31:18.359168 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:18.358904 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:31:18.359168 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:18.358934 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:31:18.437245 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:18.437215 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" event={"ID":"89db98c1-0628-4d66-a459-c80ad509c607","Type":"ContainerStarted","Data":"b8c97174156231ae4045bca8f2e313a60e8879c0c447f075fd7f18f6a48ac1be"} Jul 06 23:31:18.457954 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:18.457898 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" podStartSLOduration=0.82794953 podStartE2EDuration="3.457882567s" totalImagesPullingTime="2.629933037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:31:18.456271208 +0000 UTC m=+389.170325780" watchObservedRunningTime="2026-07-06 23:31:18.457882567 +0000 UTC m=+389.171937137" Jul 06 23:31:18.565522 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:18.565482 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:19.569805 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:19.569773 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:20.446174 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:20.446145 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:20.447239 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:20.447221 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48kwjn" Jul 06 23:31:58.691370 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:58.691339 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bfkdb"] Jul 06 23:31:58.700993 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:58.700966 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bfkdb" Jul 06 23:31:58.704017 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:58.703995 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-k7242\"" Jul 06 23:31:58.704195 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:58.704013 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 23:31:58.704459 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:58.704058 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 23:31:58.705243 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:58.705215 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bfkdb"] Jul 06 23:31:58.770873 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:58.770838 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2nlm\" (UniqueName: \"kubernetes.io/projected/6bacdb56-4916-438c-8fdb-65bc80f1e830-kube-api-access-z2nlm\") pod \"kuadrant-operator-catalog-bfkdb\" (UID: \"6bacdb56-4916-438c-8fdb-65bc80f1e830\") " pod="kuadrant-system/kuadrant-operator-catalog-bfkdb" Jul 06 23:31:58.871278 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:58.871242 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2nlm\" (UniqueName: \"kubernetes.io/projected/6bacdb56-4916-438c-8fdb-65bc80f1e830-kube-api-access-z2nlm\") pod \"kuadrant-operator-catalog-bfkdb\" (UID: \"6bacdb56-4916-438c-8fdb-65bc80f1e830\") " pod="kuadrant-system/kuadrant-operator-catalog-bfkdb" Jul 06 23:31:58.886649 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:58.886624 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2nlm\" (UniqueName: \"kubernetes.io/projected/6bacdb56-4916-438c-8fdb-65bc80f1e830-kube-api-access-z2nlm\") pod \"kuadrant-operator-catalog-bfkdb\" (UID: \"6bacdb56-4916-438c-8fdb-65bc80f1e830\") " pod="kuadrant-system/kuadrant-operator-catalog-bfkdb" Jul 06 23:31:59.012335 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:59.012251 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bfkdb" Jul 06 23:31:59.063757 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:59.063707 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bfkdb"] Jul 06 23:31:59.142630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:59.142500 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bfkdb"] Jul 06 23:31:59.145439 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:31:59.145414 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bacdb56_4916_438c_8fdb_65bc80f1e830.slice/crio-14d210a6d736fcc659c28aac65352f8d12f48fa8e5b228340c9c884e99d2d77f WatchSource:0}: Error finding container 14d210a6d736fcc659c28aac65352f8d12f48fa8e5b228340c9c884e99d2d77f: Status 404 returned error can't find the container with id 14d210a6d736fcc659c28aac65352f8d12f48fa8e5b228340c9c884e99d2d77f Jul 06 23:31:59.275189 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:59.275115 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-g8qcs"] Jul 06 23:31:59.280123 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:59.280104 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-g8qcs" Jul 06 23:31:59.285954 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:59.285927 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-g8qcs"] Jul 06 23:31:59.375219 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:59.375182 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk7dg\" (UniqueName: \"kubernetes.io/projected/e74cb8a1-cf4d-474a-9cb1-5cb378dd680e-kube-api-access-zk7dg\") pod \"kuadrant-operator-catalog-g8qcs\" (UID: \"e74cb8a1-cf4d-474a-9cb1-5cb378dd680e\") " pod="kuadrant-system/kuadrant-operator-catalog-g8qcs" Jul 06 23:31:59.476247 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:59.476208 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk7dg\" (UniqueName: \"kubernetes.io/projected/e74cb8a1-cf4d-474a-9cb1-5cb378dd680e-kube-api-access-zk7dg\") pod \"kuadrant-operator-catalog-g8qcs\" (UID: \"e74cb8a1-cf4d-474a-9cb1-5cb378dd680e\") " pod="kuadrant-system/kuadrant-operator-catalog-g8qcs" Jul 06 23:31:59.484255 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:59.484226 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk7dg\" (UniqueName: \"kubernetes.io/projected/e74cb8a1-cf4d-474a-9cb1-5cb378dd680e-kube-api-access-zk7dg\") pod \"kuadrant-operator-catalog-g8qcs\" (UID: \"e74cb8a1-cf4d-474a-9cb1-5cb378dd680e\") " pod="kuadrant-system/kuadrant-operator-catalog-g8qcs" Jul 06 23:31:59.590111 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:59.590035 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-g8qcs" Jul 06 23:31:59.606949 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:59.606902 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bfkdb" event={"ID":"6bacdb56-4916-438c-8fdb-65bc80f1e830","Type":"ContainerStarted","Data":"14d210a6d736fcc659c28aac65352f8d12f48fa8e5b228340c9c884e99d2d77f"} Jul 06 23:31:59.735287 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:31:59.735256 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-g8qcs"] Jul 06 23:31:59.737251 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:31:59.737224 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74cb8a1_cf4d_474a_9cb1_5cb378dd680e.slice/crio-563eb17ebc107d33af8a94d46bf0173fac483bab465afb219f2a8dbf9cd67c25 WatchSource:0}: Error finding container 563eb17ebc107d33af8a94d46bf0173fac483bab465afb219f2a8dbf9cd67c25: Status 404 returned error can't find the container with id 563eb17ebc107d33af8a94d46bf0173fac483bab465afb219f2a8dbf9cd67c25 Jul 06 23:32:00.614411 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:00.614372 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-g8qcs" event={"ID":"e74cb8a1-cf4d-474a-9cb1-5cb378dd680e","Type":"ContainerStarted","Data":"563eb17ebc107d33af8a94d46bf0173fac483bab465afb219f2a8dbf9cd67c25"} Jul 06 23:32:01.620068 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:01.620029 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-g8qcs" event={"ID":"e74cb8a1-cf4d-474a-9cb1-5cb378dd680e","Type":"ContainerStarted","Data":"ad61ff7f3864bd934bb93c4a4ac0d204e4d5c904bcb1c93379795d613e42b487"} Jul 06 23:32:01.621554 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:01.621523 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bfkdb" event={"ID":"6bacdb56-4916-438c-8fdb-65bc80f1e830","Type":"ContainerStarted","Data":"48d92ba67fc2d315d8f86e0e10a04a6c548beb1b3ba6b0be6a6277cfe6a3aa03"} Jul 06 23:32:01.621674 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:01.621579 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-bfkdb" podUID="6bacdb56-4916-438c-8fdb-65bc80f1e830" containerName="registry-server" containerID="cri-o://48d92ba67fc2d315d8f86e0e10a04a6c548beb1b3ba6b0be6a6277cfe6a3aa03" gracePeriod=2 Jul 06 23:32:01.638132 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:01.638092 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-g8qcs" podStartSLOduration=1.345197442 podStartE2EDuration="2.638077544s" totalImagesPullingTime="1.292880102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:31:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:32:01.636132078 +0000 UTC m=+432.350186658" watchObservedRunningTime="2026-07-06 23:32:01.638077544 +0000 UTC m=+432.352132113" Jul 06 23:32:01.650763 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:01.650643 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-bfkdb" podStartSLOduration=1.7693030250000001 podStartE2EDuration="3.650624972s" totalImagesPullingTime="1.881321947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:32:01.650159645 +0000 UTC m=+432.364214218" watchObservedRunningTime="2026-07-06 23:32:01.650624972 +0000 UTC m=+432.364679542" Jul 06 23:32:01.866173 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:01.866148 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bfkdb" Jul 06 23:32:01.996447 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:01.996412 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2nlm\" (UniqueName: \"kubernetes.io/projected/6bacdb56-4916-438c-8fdb-65bc80f1e830-kube-api-access-z2nlm\") pod \"6bacdb56-4916-438c-8fdb-65bc80f1e830\" (UID: \"6bacdb56-4916-438c-8fdb-65bc80f1e830\") " Jul 06 23:32:01.998655 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:01.998627 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bacdb56-4916-438c-8fdb-65bc80f1e830-kube-api-access-z2nlm" (OuterVolumeSpecName: "kube-api-access-z2nlm") pod "6bacdb56-4916-438c-8fdb-65bc80f1e830" (UID: "6bacdb56-4916-438c-8fdb-65bc80f1e830"). InnerVolumeSpecName "kube-api-access-z2nlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:32:02.097135 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:02.097106 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2nlm\" (UniqueName: \"kubernetes.io/projected/6bacdb56-4916-438c-8fdb-65bc80f1e830-kube-api-access-z2nlm\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:32:02.626733 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:02.626697 2571 generic.go:356] "Generic (PLEG): container finished" podID="6bacdb56-4916-438c-8fdb-65bc80f1e830" containerID="48d92ba67fc2d315d8f86e0e10a04a6c548beb1b3ba6b0be6a6277cfe6a3aa03" exitCode=0 Jul 06 23:32:02.627189 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:02.626765 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bfkdb" Jul 06 23:32:02.627189 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:02.626792 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bfkdb" event={"ID":"6bacdb56-4916-438c-8fdb-65bc80f1e830","Type":"ContainerDied","Data":"48d92ba67fc2d315d8f86e0e10a04a6c548beb1b3ba6b0be6a6277cfe6a3aa03"} Jul 06 23:32:02.627189 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:02.626848 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bfkdb" event={"ID":"6bacdb56-4916-438c-8fdb-65bc80f1e830","Type":"ContainerDied","Data":"14d210a6d736fcc659c28aac65352f8d12f48fa8e5b228340c9c884e99d2d77f"} Jul 06 23:32:02.627189 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:02.626880 2571 scope.go:117] "RemoveContainer" containerID="48d92ba67fc2d315d8f86e0e10a04a6c548beb1b3ba6b0be6a6277cfe6a3aa03" Jul 06 23:32:02.636067 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:02.636048 2571 scope.go:117] "RemoveContainer" containerID="48d92ba67fc2d315d8f86e0e10a04a6c548beb1b3ba6b0be6a6277cfe6a3aa03" Jul 06 23:32:02.636329 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:02.636310 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d92ba67fc2d315d8f86e0e10a04a6c548beb1b3ba6b0be6a6277cfe6a3aa03\": container with ID starting with 48d92ba67fc2d315d8f86e0e10a04a6c548beb1b3ba6b0be6a6277cfe6a3aa03 not found: ID does not exist" containerID="48d92ba67fc2d315d8f86e0e10a04a6c548beb1b3ba6b0be6a6277cfe6a3aa03" Jul 06 23:32:02.636389 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:02.636338 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d92ba67fc2d315d8f86e0e10a04a6c548beb1b3ba6b0be6a6277cfe6a3aa03"} err="failed to get container status \"48d92ba67fc2d315d8f86e0e10a04a6c548beb1b3ba6b0be6a6277cfe6a3aa03\": rpc error: code = NotFound desc = could not find container \"48d92ba67fc2d315d8f86e0e10a04a6c548beb1b3ba6b0be6a6277cfe6a3aa03\": container with ID starting with 48d92ba67fc2d315d8f86e0e10a04a6c548beb1b3ba6b0be6a6277cfe6a3aa03 not found: ID does not exist" Jul 06 23:32:02.647064 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:02.647036 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bfkdb"] Jul 06 23:32:02.650943 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:02.650925 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bfkdb"] Jul 06 23:32:03.785248 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:03.785213 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bacdb56-4916-438c-8fdb-65bc80f1e830" path="/var/lib/kubelet/pods/6bacdb56-4916-438c-8fdb-65bc80f1e830/volumes" Jul 06 23:32:09.590159 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:09.590125 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-g8qcs" Jul 06 23:32:09.590541 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:09.590183 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-g8qcs" Jul 06 23:32:09.612996 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:09.612973 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-g8qcs" Jul 06 23:32:09.678436 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:09.678413 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-g8qcs" Jul 06 23:32:14.094976 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.094904 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m"] Jul 06 23:32:14.095307 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.095175 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bacdb56-4916-438c-8fdb-65bc80f1e830" containerName="registry-server" Jul 06 23:32:14.095307 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.095185 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bacdb56-4916-438c-8fdb-65bc80f1e830" containerName="registry-server" Jul 06 23:32:14.095307 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.095238 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bacdb56-4916-438c-8fdb-65bc80f1e830" containerName="registry-server" Jul 06 23:32:14.098060 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.098046 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m" Jul 06 23:32:14.101040 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.101015 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zn66c\"" Jul 06 23:32:14.115347 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.115323 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m"] Jul 06 23:32:14.184056 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.184035 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71e387a9-a244-4335-9fdf-38c8c24cdbbf-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m\" (UID: \"71e387a9-a244-4335-9fdf-38c8c24cdbbf\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m" Jul 06 23:32:14.184187 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.184074 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71e387a9-a244-4335-9fdf-38c8c24cdbbf-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m\" (UID: \"71e387a9-a244-4335-9fdf-38c8c24cdbbf\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m" Jul 06 23:32:14.184187 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.184107 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrjhx\" (UniqueName: \"kubernetes.io/projected/71e387a9-a244-4335-9fdf-38c8c24cdbbf-kube-api-access-lrjhx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m\" (UID: \"71e387a9-a244-4335-9fdf-38c8c24cdbbf\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m" Jul 06 23:32:14.284773 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.284755 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71e387a9-a244-4335-9fdf-38c8c24cdbbf-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m\" (UID: \"71e387a9-a244-4335-9fdf-38c8c24cdbbf\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m" Jul 06 23:32:14.284871 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.284795 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrjhx\" (UniqueName: \"kubernetes.io/projected/71e387a9-a244-4335-9fdf-38c8c24cdbbf-kube-api-access-lrjhx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m\" (UID: \"71e387a9-a244-4335-9fdf-38c8c24cdbbf\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m" Jul 06 23:32:14.284871 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.284830 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71e387a9-a244-4335-9fdf-38c8c24cdbbf-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m\" (UID: \"71e387a9-a244-4335-9fdf-38c8c24cdbbf\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m" Jul 06 23:32:14.285079 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.285062 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71e387a9-a244-4335-9fdf-38c8c24cdbbf-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m\" (UID: \"71e387a9-a244-4335-9fdf-38c8c24cdbbf\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m" Jul 06 23:32:14.285117 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.285094 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71e387a9-a244-4335-9fdf-38c8c24cdbbf-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m\" (UID: \"71e387a9-a244-4335-9fdf-38c8c24cdbbf\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m" Jul 06 23:32:14.293748 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.293703 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrjhx\" (UniqueName: \"kubernetes.io/projected/71e387a9-a244-4335-9fdf-38c8c24cdbbf-kube-api-access-lrjhx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m\" (UID: \"71e387a9-a244-4335-9fdf-38c8c24cdbbf\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m" Jul 06 23:32:14.407306 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.407257 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m" Jul 06 23:32:14.498578 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.498544 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q"] Jul 06 23:32:14.502922 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.502903 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q" Jul 06 23:32:14.512472 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.512430 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q"] Jul 06 23:32:14.559616 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.559599 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m"] Jul 06 23:32:14.561446 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:32:14.561423 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e387a9_a244_4335_9fdf_38c8c24cdbbf.slice/crio-b3683d0c61ac181906611eb6e1f02e9090199e562c977476ec0c4f4cbcc5ee2d WatchSource:0}: Error finding container b3683d0c61ac181906611eb6e1f02e9090199e562c977476ec0c4f4cbcc5ee2d: Status 404 returned error can't find the container with id b3683d0c61ac181906611eb6e1f02e9090199e562c977476ec0c4f4cbcc5ee2d Jul 06 23:32:14.587630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.587604 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnvzh\" (UniqueName: \"kubernetes.io/projected/eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8-kube-api-access-fnvzh\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q\" (UID: \"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q" Jul 06 23:32:14.587733 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.587661 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q\" (UID: \"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q" Jul 06 23:32:14.587776 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.587727 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q\" (UID: \"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q" Jul 06 23:32:14.680367 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.680339 2571 generic.go:356] "Generic (PLEG): container finished" podID="71e387a9-a244-4335-9fdf-38c8c24cdbbf" containerID="7f92193ee7e7301b2b0946977c079ff95d831433042f22ee60a8d9f6a15465ca" exitCode=0 Jul 06 23:32:14.680477 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.680416 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m" event={"ID":"71e387a9-a244-4335-9fdf-38c8c24cdbbf","Type":"ContainerDied","Data":"7f92193ee7e7301b2b0946977c079ff95d831433042f22ee60a8d9f6a15465ca"} Jul 06 23:32:14.680477 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.680471 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m" event={"ID":"71e387a9-a244-4335-9fdf-38c8c24cdbbf","Type":"ContainerStarted","Data":"b3683d0c61ac181906611eb6e1f02e9090199e562c977476ec0c4f4cbcc5ee2d"} Jul 06 23:32:14.689208 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.689181 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q\" (UID: \"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q" Jul 06 23:32:14.689300 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.689227 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q\" (UID: \"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q" Jul 06 23:32:14.689300 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.689292 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnvzh\" (UniqueName: \"kubernetes.io/projected/eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8-kube-api-access-fnvzh\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q\" (UID: \"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q" Jul 06 23:32:14.689554 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.689534 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q\" (UID: \"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q" Jul 06 23:32:14.689609 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.689551 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q\" (UID: \"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q" Jul 06 23:32:14.698565 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.698528 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnvzh\" (UniqueName: \"kubernetes.io/projected/eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8-kube-api-access-fnvzh\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q\" (UID: \"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q" Jul 06 23:32:14.814341 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.814320 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q" Jul 06 23:32:14.887675 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.887650 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5"] Jul 06 23:32:14.892139 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.892120 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5" Jul 06 23:32:14.904320 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.904298 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5"] Jul 06 23:32:14.944757 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.944593 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q"] Jul 06 23:32:14.947288 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:32:14.947270 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeccc2e39_66b4_4cd3_93e9_97fdbdfc41f8.slice/crio-fdbb7dcfd554f09cf9ed515eef5422b61675c071c18fd57a2f1aaadce96685c3 WatchSource:0}: Error finding container fdbb7dcfd554f09cf9ed515eef5422b61675c071c18fd57a2f1aaadce96685c3: Status 404 returned error can't find the container with id fdbb7dcfd554f09cf9ed515eef5422b61675c071c18fd57a2f1aaadce96685c3 Jul 06 23:32:14.991561 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.991540 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7320a85d-c52d-4f7a-bfa8-467cfa332ed3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5\" (UID: \"7320a85d-c52d-4f7a-bfa8-467cfa332ed3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5" Jul 06 23:32:14.991644 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.991573 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jf6p\" (UniqueName: \"kubernetes.io/projected/7320a85d-c52d-4f7a-bfa8-467cfa332ed3-kube-api-access-9jf6p\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5\" (UID: \"7320a85d-c52d-4f7a-bfa8-467cfa332ed3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5" Jul 06 23:32:14.991644 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:14.991593 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7320a85d-c52d-4f7a-bfa8-467cfa332ed3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5\" (UID: \"7320a85d-c52d-4f7a-bfa8-467cfa332ed3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5" Jul 06 23:32:15.092172 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.092145 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7320a85d-c52d-4f7a-bfa8-467cfa332ed3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5\" (UID: \"7320a85d-c52d-4f7a-bfa8-467cfa332ed3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5" Jul 06 23:32:15.092294 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.092185 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jf6p\" (UniqueName: \"kubernetes.io/projected/7320a85d-c52d-4f7a-bfa8-467cfa332ed3-kube-api-access-9jf6p\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5\" (UID: \"7320a85d-c52d-4f7a-bfa8-467cfa332ed3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5" Jul 06 23:32:15.092294 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.092204 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7320a85d-c52d-4f7a-bfa8-467cfa332ed3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5\" (UID: \"7320a85d-c52d-4f7a-bfa8-467cfa332ed3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5" Jul 06 23:32:15.097303 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.092975 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7320a85d-c52d-4f7a-bfa8-467cfa332ed3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5\" (UID: \"7320a85d-c52d-4f7a-bfa8-467cfa332ed3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5" Jul 06 23:32:15.097303 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.093028 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7320a85d-c52d-4f7a-bfa8-467cfa332ed3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5\" (UID: \"7320a85d-c52d-4f7a-bfa8-467cfa332ed3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5" Jul 06 23:32:15.102372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.102353 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jf6p\" (UniqueName: \"kubernetes.io/projected/7320a85d-c52d-4f7a-bfa8-467cfa332ed3-kube-api-access-9jf6p\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5\" (UID: \"7320a85d-c52d-4f7a-bfa8-467cfa332ed3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5" Jul 06 23:32:15.203555 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.203530 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5" Jul 06 23:32:15.292533 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.292505 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx"] Jul 06 23:32:15.295978 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.295955 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx" Jul 06 23:32:15.307367 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.306001 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx"] Jul 06 23:32:15.343866 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.343838 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5"] Jul 06 23:32:15.354677 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:32:15.354649 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7320a85d_c52d_4f7a_bfa8_467cfa332ed3.slice/crio-b4ea04fe8a2e969b28a66cccd2556e9e65c82be4b2fdeaa4146cdab840d3a702 WatchSource:0}: Error finding container b4ea04fe8a2e969b28a66cccd2556e9e65c82be4b2fdeaa4146cdab840d3a702: Status 404 returned error can't find the container with id b4ea04fe8a2e969b28a66cccd2556e9e65c82be4b2fdeaa4146cdab840d3a702 Jul 06 23:32:15.395587 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.395562 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bae9384-1e9a-4da5-abb0-7b2d61d4732c-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx\" (UID: \"1bae9384-1e9a-4da5-abb0-7b2d61d4732c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx" Jul 06 23:32:15.395681 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.395606 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bae9384-1e9a-4da5-abb0-7b2d61d4732c-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx\" (UID: \"1bae9384-1e9a-4da5-abb0-7b2d61d4732c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx" Jul 06 23:32:15.395681 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.395645 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv62d\" (UniqueName: \"kubernetes.io/projected/1bae9384-1e9a-4da5-abb0-7b2d61d4732c-kube-api-access-xv62d\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx\" (UID: \"1bae9384-1e9a-4da5-abb0-7b2d61d4732c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx" Jul 06 23:32:15.496942 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.496905 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bae9384-1e9a-4da5-abb0-7b2d61d4732c-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx\" (UID: \"1bae9384-1e9a-4da5-abb0-7b2d61d4732c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx" Jul 06 23:32:15.497055 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.496964 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bae9384-1e9a-4da5-abb0-7b2d61d4732c-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx\" (UID: \"1bae9384-1e9a-4da5-abb0-7b2d61d4732c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx" Jul 06 23:32:15.497055 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.497008 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv62d\" (UniqueName: \"kubernetes.io/projected/1bae9384-1e9a-4da5-abb0-7b2d61d4732c-kube-api-access-xv62d\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx\" (UID: \"1bae9384-1e9a-4da5-abb0-7b2d61d4732c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx" Jul 06 23:32:15.497335 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.497314 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bae9384-1e9a-4da5-abb0-7b2d61d4732c-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx\" (UID: \"1bae9384-1e9a-4da5-abb0-7b2d61d4732c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx" Jul 06 23:32:15.497367 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.497322 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bae9384-1e9a-4da5-abb0-7b2d61d4732c-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx\" (UID: \"1bae9384-1e9a-4da5-abb0-7b2d61d4732c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx" Jul 06 23:32:15.506383 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.506353 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv62d\" (UniqueName: \"kubernetes.io/projected/1bae9384-1e9a-4da5-abb0-7b2d61d4732c-kube-api-access-xv62d\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx\" (UID: \"1bae9384-1e9a-4da5-abb0-7b2d61d4732c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx" Jul 06 23:32:15.661062 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.661027 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx" Jul 06 23:32:15.686669 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.686640 2571 generic.go:356] "Generic (PLEG): container finished" podID="7320a85d-c52d-4f7a-bfa8-467cfa332ed3" containerID="678e95b5be387e3f4a02a4638bce6d23a4cb52d836bf00e666a03f1a57a19457" exitCode=0 Jul 06 23:32:15.686804 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.686735 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5" event={"ID":"7320a85d-c52d-4f7a-bfa8-467cfa332ed3","Type":"ContainerDied","Data":"678e95b5be387e3f4a02a4638bce6d23a4cb52d836bf00e666a03f1a57a19457"} Jul 06 23:32:15.686804 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.686792 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5" event={"ID":"7320a85d-c52d-4f7a-bfa8-467cfa332ed3","Type":"ContainerStarted","Data":"b4ea04fe8a2e969b28a66cccd2556e9e65c82be4b2fdeaa4146cdab840d3a702"} Jul 06 23:32:15.688350 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.688326 2571 generic.go:356] "Generic (PLEG): container finished" podID="eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8" containerID="bc735cdda246884ba0b1cd98a50de01c62e544ec27c1f04c6dc5362796b7bfad" exitCode=0 Jul 06 23:32:15.688463 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.688409 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q" event={"ID":"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8","Type":"ContainerDied","Data":"bc735cdda246884ba0b1cd98a50de01c62e544ec27c1f04c6dc5362796b7bfad"} Jul 06 23:32:15.688518 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.688460 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q" event={"ID":"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8","Type":"ContainerStarted","Data":"fdbb7dcfd554f09cf9ed515eef5422b61675c071c18fd57a2f1aaadce96685c3"} Jul 06 23:32:15.690494 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.690424 2571 generic.go:356] "Generic (PLEG): container finished" podID="71e387a9-a244-4335-9fdf-38c8c24cdbbf" containerID="e996c8cd2f0fbc523261988ba97ed426dc81e8393b71906912c0c7687fffe97f" exitCode=0 Jul 06 23:32:15.690494 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.690449 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m" event={"ID":"71e387a9-a244-4335-9fdf-38c8c24cdbbf","Type":"ContainerDied","Data":"e996c8cd2f0fbc523261988ba97ed426dc81e8393b71906912c0c7687fffe97f"} Jul 06 23:32:15.799697 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:15.799668 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx"] Jul 06 23:32:15.801302 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:32:15.801284 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bae9384_1e9a_4da5_abb0_7b2d61d4732c.slice/crio-d1e4735b68fd822b6ce1fa20d703df1eea474cfc236f47a15ef49391d3d98d2b WatchSource:0}: Error finding container d1e4735b68fd822b6ce1fa20d703df1eea474cfc236f47a15ef49391d3d98d2b: Status 404 returned error can't find the container with id d1e4735b68fd822b6ce1fa20d703df1eea474cfc236f47a15ef49391d3d98d2b Jul 06 23:32:16.697086 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:16.697055 2571 generic.go:356] "Generic (PLEG): container finished" podID="71e387a9-a244-4335-9fdf-38c8c24cdbbf" containerID="d22a73c514fda4c4a25dd2ed10a6d360c368d735792b092ab200abb1d2f20350" exitCode=0 Jul 06 23:32:16.697468 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:16.697127 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m" event={"ID":"71e387a9-a244-4335-9fdf-38c8c24cdbbf","Type":"ContainerDied","Data":"d22a73c514fda4c4a25dd2ed10a6d360c368d735792b092ab200abb1d2f20350"} Jul 06 23:32:16.698535 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:16.698515 2571 generic.go:356] "Generic (PLEG): container finished" podID="1bae9384-1e9a-4da5-abb0-7b2d61d4732c" containerID="cd1f1348e8f2a31959412fc0f5d9b095879a264c9f6b05c34c6ff9d042f12b1c" exitCode=0 Jul 06 23:32:16.698639 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:16.698605 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx" event={"ID":"1bae9384-1e9a-4da5-abb0-7b2d61d4732c","Type":"ContainerDied","Data":"cd1f1348e8f2a31959412fc0f5d9b095879a264c9f6b05c34c6ff9d042f12b1c"} Jul 06 23:32:16.698687 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:16.698653 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx" event={"ID":"1bae9384-1e9a-4da5-abb0-7b2d61d4732c","Type":"ContainerStarted","Data":"d1e4735b68fd822b6ce1fa20d703df1eea474cfc236f47a15ef49391d3d98d2b"} Jul 06 23:32:16.700393 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:16.700373 2571 generic.go:356] "Generic (PLEG): container finished" podID="7320a85d-c52d-4f7a-bfa8-467cfa332ed3" containerID="8b5f88fdd6bb65f8fdacad1f3e88ceaeeb218b521f921606a4099c147e893976" exitCode=0 Jul 06 23:32:16.700509 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:16.700458 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5" event={"ID":"7320a85d-c52d-4f7a-bfa8-467cfa332ed3","Type":"ContainerDied","Data":"8b5f88fdd6bb65f8fdacad1f3e88ceaeeb218b521f921606a4099c147e893976"} Jul 06 23:32:16.702449 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:16.702417 2571 generic.go:356] "Generic (PLEG): container finished" podID="eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8" containerID="94c6e0d9894bbede80983abe12eef65e4c648f0ec0fed7ce97befc8443cfffb2" exitCode=0 Jul 06 23:32:16.702543 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:16.702465 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q" event={"ID":"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8","Type":"ContainerDied","Data":"94c6e0d9894bbede80983abe12eef65e4c648f0ec0fed7ce97befc8443cfffb2"} Jul 06 23:32:17.710455 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:17.710422 2571 generic.go:356] "Generic (PLEG): container finished" podID="7320a85d-c52d-4f7a-bfa8-467cfa332ed3" containerID="66ba95936eca7842d97e7f5a2efa7b3a5f7e8b53efa7447f95103834132a291e" exitCode=0 Jul 06 23:32:17.710873 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:17.710491 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5" event={"ID":"7320a85d-c52d-4f7a-bfa8-467cfa332ed3","Type":"ContainerDied","Data":"66ba95936eca7842d97e7f5a2efa7b3a5f7e8b53efa7447f95103834132a291e"} Jul 06 23:32:17.712462 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:17.712427 2571 generic.go:356] "Generic (PLEG): container finished" podID="eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8" containerID="9c76d649a252e3eab16277b2c692c7726c111246d8593cd399708595235e7a95" exitCode=0 Jul 06 23:32:17.712575 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:17.712508 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q" event={"ID":"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8","Type":"ContainerDied","Data":"9c76d649a252e3eab16277b2c692c7726c111246d8593cd399708595235e7a95"} Jul 06 23:32:17.714193 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:17.714172 2571 generic.go:356] "Generic (PLEG): container finished" podID="1bae9384-1e9a-4da5-abb0-7b2d61d4732c" containerID="57a18572087a302e0b99ead912972c0f18c8e1e13662a37c289c3547e03bd31b" exitCode=0 Jul 06 23:32:17.714316 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:17.714223 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx" event={"ID":"1bae9384-1e9a-4da5-abb0-7b2d61d4732c","Type":"ContainerDied","Data":"57a18572087a302e0b99ead912972c0f18c8e1e13662a37c289c3547e03bd31b"} Jul 06 23:32:17.851542 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:17.851520 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m" Jul 06 23:32:17.919319 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:17.919294 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71e387a9-a244-4335-9fdf-38c8c24cdbbf-util\") pod \"71e387a9-a244-4335-9fdf-38c8c24cdbbf\" (UID: \"71e387a9-a244-4335-9fdf-38c8c24cdbbf\") " Jul 06 23:32:17.919470 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:17.919343 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrjhx\" (UniqueName: \"kubernetes.io/projected/71e387a9-a244-4335-9fdf-38c8c24cdbbf-kube-api-access-lrjhx\") pod \"71e387a9-a244-4335-9fdf-38c8c24cdbbf\" (UID: \"71e387a9-a244-4335-9fdf-38c8c24cdbbf\") " Jul 06 23:32:17.919470 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:17.919375 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71e387a9-a244-4335-9fdf-38c8c24cdbbf-bundle\") pod \"71e387a9-a244-4335-9fdf-38c8c24cdbbf\" (UID: \"71e387a9-a244-4335-9fdf-38c8c24cdbbf\") " Jul 06 23:32:17.919886 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:17.919861 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71e387a9-a244-4335-9fdf-38c8c24cdbbf-bundle" (OuterVolumeSpecName: "bundle") pod "71e387a9-a244-4335-9fdf-38c8c24cdbbf" (UID: "71e387a9-a244-4335-9fdf-38c8c24cdbbf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:32:17.921434 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:17.921411 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e387a9-a244-4335-9fdf-38c8c24cdbbf-kube-api-access-lrjhx" (OuterVolumeSpecName: "kube-api-access-lrjhx") pod "71e387a9-a244-4335-9fdf-38c8c24cdbbf" (UID: "71e387a9-a244-4335-9fdf-38c8c24cdbbf"). InnerVolumeSpecName "kube-api-access-lrjhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:32:17.925017 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:17.924997 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71e387a9-a244-4335-9fdf-38c8c24cdbbf-util" (OuterVolumeSpecName: "util") pod "71e387a9-a244-4335-9fdf-38c8c24cdbbf" (UID: "71e387a9-a244-4335-9fdf-38c8c24cdbbf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:32:18.020655 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:18.020593 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71e387a9-a244-4335-9fdf-38c8c24cdbbf-util\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:32:18.020655 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:18.020615 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrjhx\" (UniqueName: \"kubernetes.io/projected/71e387a9-a244-4335-9fdf-38c8c24cdbbf-kube-api-access-lrjhx\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:32:18.020655 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:18.020629 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71e387a9-a244-4335-9fdf-38c8c24cdbbf-bundle\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:32:18.721737 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:18.721691 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m" Jul 06 23:32:18.721737 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:18.721694 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m" event={"ID":"71e387a9-a244-4335-9fdf-38c8c24cdbbf","Type":"ContainerDied","Data":"b3683d0c61ac181906611eb6e1f02e9090199e562c977476ec0c4f4cbcc5ee2d"} Jul 06 23:32:18.722248 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:18.721768 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3683d0c61ac181906611eb6e1f02e9090199e562c977476ec0c4f4cbcc5ee2d" Jul 06 23:32:18.723782 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:18.723756 2571 generic.go:356] "Generic (PLEG): container finished" podID="1bae9384-1e9a-4da5-abb0-7b2d61d4732c" containerID="a9765abe54a85b29f1e718026a15f001ab33cfed0656035899c0c0657523dfc4" exitCode=0 Jul 06 23:32:18.723919 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:18.723836 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx" event={"ID":"1bae9384-1e9a-4da5-abb0-7b2d61d4732c","Type":"ContainerDied","Data":"a9765abe54a85b29f1e718026a15f001ab33cfed0656035899c0c0657523dfc4"} Jul 06 23:32:18.850880 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:18.850858 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5" Jul 06 23:32:18.882755 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:18.882734 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q" Jul 06 23:32:18.926923 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:18.926901 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jf6p\" (UniqueName: \"kubernetes.io/projected/7320a85d-c52d-4f7a-bfa8-467cfa332ed3-kube-api-access-9jf6p\") pod \"7320a85d-c52d-4f7a-bfa8-467cfa332ed3\" (UID: \"7320a85d-c52d-4f7a-bfa8-467cfa332ed3\") " Jul 06 23:32:18.927028 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:18.926944 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7320a85d-c52d-4f7a-bfa8-467cfa332ed3-util\") pod \"7320a85d-c52d-4f7a-bfa8-467cfa332ed3\" (UID: \"7320a85d-c52d-4f7a-bfa8-467cfa332ed3\") " Jul 06 23:32:18.927028 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:18.927014 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7320a85d-c52d-4f7a-bfa8-467cfa332ed3-bundle\") pod \"7320a85d-c52d-4f7a-bfa8-467cfa332ed3\" (UID: \"7320a85d-c52d-4f7a-bfa8-467cfa332ed3\") " Jul 06 23:32:18.927490 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:18.927470 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7320a85d-c52d-4f7a-bfa8-467cfa332ed3-bundle" (OuterVolumeSpecName: "bundle") pod "7320a85d-c52d-4f7a-bfa8-467cfa332ed3" (UID: "7320a85d-c52d-4f7a-bfa8-467cfa332ed3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:32:18.929180 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:18.929160 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7320a85d-c52d-4f7a-bfa8-467cfa332ed3-kube-api-access-9jf6p" (OuterVolumeSpecName: "kube-api-access-9jf6p") pod "7320a85d-c52d-4f7a-bfa8-467cfa332ed3" (UID: "7320a85d-c52d-4f7a-bfa8-467cfa332ed3"). InnerVolumeSpecName "kube-api-access-9jf6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:32:18.932293 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:18.932258 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7320a85d-c52d-4f7a-bfa8-467cfa332ed3-util" (OuterVolumeSpecName: "util") pod "7320a85d-c52d-4f7a-bfa8-467cfa332ed3" (UID: "7320a85d-c52d-4f7a-bfa8-467cfa332ed3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:32:19.028156 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.028095 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8-bundle\") pod \"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8\" (UID: \"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8\") " Jul 06 23:32:19.028253 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.028176 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8-util\") pod \"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8\" (UID: \"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8\") " Jul 06 23:32:19.028253 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.028224 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnvzh\" (UniqueName: \"kubernetes.io/projected/eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8-kube-api-access-fnvzh\") pod \"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8\" (UID: \"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8\") " Jul 06 23:32:19.028408 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.028392 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7320a85d-c52d-4f7a-bfa8-467cfa332ed3-util\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:32:19.028473 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.028414 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7320a85d-c52d-4f7a-bfa8-467cfa332ed3-bundle\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:32:19.028473 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.028428 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jf6p\" (UniqueName: \"kubernetes.io/projected/7320a85d-c52d-4f7a-bfa8-467cfa332ed3-kube-api-access-9jf6p\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:32:19.028768 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.028740 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8-bundle" (OuterVolumeSpecName: "bundle") pod "eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8" (UID: "eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:32:19.030287 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.030267 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8-kube-api-access-fnvzh" (OuterVolumeSpecName: "kube-api-access-fnvzh") pod "eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8" (UID: "eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8"). InnerVolumeSpecName "kube-api-access-fnvzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:32:19.032967 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.032946 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8-util" (OuterVolumeSpecName: "util") pod "eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8" (UID: "eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:32:19.128863 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.128839 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnvzh\" (UniqueName: \"kubernetes.io/projected/eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8-kube-api-access-fnvzh\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:32:19.128863 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.128862 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8-bundle\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:32:19.129004 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.128871 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8-util\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:32:19.729470 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.729438 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5" event={"ID":"7320a85d-c52d-4f7a-bfa8-467cfa332ed3","Type":"ContainerDied","Data":"b4ea04fe8a2e969b28a66cccd2556e9e65c82be4b2fdeaa4146cdab840d3a702"} Jul 06 23:32:19.729888 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.729482 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4ea04fe8a2e969b28a66cccd2556e9e65c82be4b2fdeaa4146cdab840d3a702" Jul 06 23:32:19.729888 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.729448 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5" Jul 06 23:32:19.731258 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.731228 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q" event={"ID":"eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8","Type":"ContainerDied","Data":"fdbb7dcfd554f09cf9ed515eef5422b61675c071c18fd57a2f1aaadce96685c3"} Jul 06 23:32:19.731376 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.731274 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdbb7dcfd554f09cf9ed515eef5422b61675c071c18fd57a2f1aaadce96685c3" Jul 06 23:32:19.731419 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.731399 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q" Jul 06 23:32:19.853969 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.853944 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx" Jul 06 23:32:19.935528 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.935498 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bae9384-1e9a-4da5-abb0-7b2d61d4732c-util\") pod \"1bae9384-1e9a-4da5-abb0-7b2d61d4732c\" (UID: \"1bae9384-1e9a-4da5-abb0-7b2d61d4732c\") " Jul 06 23:32:19.935528 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.935529 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv62d\" (UniqueName: \"kubernetes.io/projected/1bae9384-1e9a-4da5-abb0-7b2d61d4732c-kube-api-access-xv62d\") pod \"1bae9384-1e9a-4da5-abb0-7b2d61d4732c\" (UID: \"1bae9384-1e9a-4da5-abb0-7b2d61d4732c\") " Jul 06 23:32:19.935685 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.935554 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bae9384-1e9a-4da5-abb0-7b2d61d4732c-bundle\") pod \"1bae9384-1e9a-4da5-abb0-7b2d61d4732c\" (UID: \"1bae9384-1e9a-4da5-abb0-7b2d61d4732c\") " Jul 06 23:32:19.936087 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.936060 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bae9384-1e9a-4da5-abb0-7b2d61d4732c-bundle" (OuterVolumeSpecName: "bundle") pod "1bae9384-1e9a-4da5-abb0-7b2d61d4732c" (UID: "1bae9384-1e9a-4da5-abb0-7b2d61d4732c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:32:19.937775 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.937756 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bae9384-1e9a-4da5-abb0-7b2d61d4732c-kube-api-access-xv62d" (OuterVolumeSpecName: "kube-api-access-xv62d") pod "1bae9384-1e9a-4da5-abb0-7b2d61d4732c" (UID: "1bae9384-1e9a-4da5-abb0-7b2d61d4732c"). InnerVolumeSpecName "kube-api-access-xv62d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:32:19.941073 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:19.941053 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bae9384-1e9a-4da5-abb0-7b2d61d4732c-util" (OuterVolumeSpecName: "util") pod "1bae9384-1e9a-4da5-abb0-7b2d61d4732c" (UID: "1bae9384-1e9a-4da5-abb0-7b2d61d4732c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:32:20.036385 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:20.036325 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bae9384-1e9a-4da5-abb0-7b2d61d4732c-bundle\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:32:20.036385 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:20.036348 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bae9384-1e9a-4da5-abb0-7b2d61d4732c-util\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:32:20.036385 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:20.036362 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv62d\" (UniqueName: \"kubernetes.io/projected/1bae9384-1e9a-4da5-abb0-7b2d61d4732c-kube-api-access-xv62d\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:32:20.737993 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:20.737964 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx" Jul 06 23:32:20.737993 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:20.737987 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx" event={"ID":"1bae9384-1e9a-4da5-abb0-7b2d61d4732c","Type":"ContainerDied","Data":"d1e4735b68fd822b6ce1fa20d703df1eea474cfc236f47a15ef49391d3d98d2b"} Jul 06 23:32:20.738475 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:20.738031 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1e4735b68fd822b6ce1fa20d703df1eea474cfc236f47a15ef49391d3d98d2b" Jul 06 23:32:30.344959 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.344924 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzxcf"] Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345233 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e387a9-a244-4335-9fdf-38c8c24cdbbf" containerName="util" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345245 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e387a9-a244-4335-9fdf-38c8c24cdbbf" containerName="util" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345256 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7320a85d-c52d-4f7a-bfa8-467cfa332ed3" containerName="extract" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345266 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7320a85d-c52d-4f7a-bfa8-467cfa332ed3" containerName="extract" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345279 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bae9384-1e9a-4da5-abb0-7b2d61d4732c" containerName="util" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345285 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bae9384-1e9a-4da5-abb0-7b2d61d4732c" containerName="util" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345293 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8" containerName="util" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345298 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8" containerName="util" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345306 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8" containerName="pull" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345311 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8" containerName="pull" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345317 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8" containerName="extract" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345322 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8" containerName="extract" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345328 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7320a85d-c52d-4f7a-bfa8-467cfa332ed3" containerName="util" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345332 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7320a85d-c52d-4f7a-bfa8-467cfa332ed3" containerName="util" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345339 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7320a85d-c52d-4f7a-bfa8-467cfa332ed3" containerName="pull" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345348 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7320a85d-c52d-4f7a-bfa8-467cfa332ed3" containerName="pull" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345357 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e387a9-a244-4335-9fdf-38c8c24cdbbf" containerName="extract" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345362 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e387a9-a244-4335-9fdf-38c8c24cdbbf" containerName="extract" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345370 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bae9384-1e9a-4da5-abb0-7b2d61d4732c" containerName="pull" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345375 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bae9384-1e9a-4da5-abb0-7b2d61d4732c" containerName="pull" Jul 06 23:32:30.345372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345382 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e387a9-a244-4335-9fdf-38c8c24cdbbf" containerName="pull" Jul 06 23:32:30.345998 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345387 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e387a9-a244-4335-9fdf-38c8c24cdbbf" containerName="pull" Jul 06 23:32:30.345998 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345393 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bae9384-1e9a-4da5-abb0-7b2d61d4732c" containerName="extract" Jul 06 23:32:30.345998 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345397 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bae9384-1e9a-4da5-abb0-7b2d61d4732c" containerName="extract" Jul 06 23:32:30.345998 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345447 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e387a9-a244-4335-9fdf-38c8c24cdbbf" containerName="extract" Jul 06 23:32:30.345998 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345455 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8" containerName="extract" Jul 06 23:32:30.345998 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345462 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7320a85d-c52d-4f7a-bfa8-467cfa332ed3" containerName="extract" Jul 06 23:32:30.345998 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.345469 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bae9384-1e9a-4da5-abb0-7b2d61d4732c" containerName="extract" Jul 06 23:32:30.351483 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.351464 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzxcf" Jul 06 23:32:30.359197 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.359181 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-q4xcv\"" Jul 06 23:32:30.359371 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.359359 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 23:32:30.364809 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.364787 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzxcf"] Jul 06 23:32:30.419374 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.419344 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsdzz\" (UniqueName: \"kubernetes.io/projected/28f9c1ca-5ec6-4f14-aaba-66dfb0f8fe0c-kube-api-access-tsdzz\") pod \"dns-operator-controller-manager-864dd78c4c-dzxcf\" (UID: \"28f9c1ca-5ec6-4f14-aaba-66dfb0f8fe0c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzxcf" Jul 06 23:32:30.520620 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.520592 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsdzz\" (UniqueName: \"kubernetes.io/projected/28f9c1ca-5ec6-4f14-aaba-66dfb0f8fe0c-kube-api-access-tsdzz\") pod \"dns-operator-controller-manager-864dd78c4c-dzxcf\" (UID: \"28f9c1ca-5ec6-4f14-aaba-66dfb0f8fe0c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzxcf" Jul 06 23:32:30.532316 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.532288 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsdzz\" (UniqueName: \"kubernetes.io/projected/28f9c1ca-5ec6-4f14-aaba-66dfb0f8fe0c-kube-api-access-tsdzz\") pod \"dns-operator-controller-manager-864dd78c4c-dzxcf\" (UID: \"28f9c1ca-5ec6-4f14-aaba-66dfb0f8fe0c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzxcf" Jul 06 23:32:30.661430 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.661376 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzxcf" Jul 06 23:32:30.795457 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:30.795436 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzxcf"] Jul 06 23:32:30.797950 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:32:30.797924 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f9c1ca_5ec6_4f14_aaba_66dfb0f8fe0c.slice/crio-61a96101e1ddf3b54fa2f1a174e5b87ba3e3d4387a6001e201fd623323b9044a WatchSource:0}: Error finding container 61a96101e1ddf3b54fa2f1a174e5b87ba3e3d4387a6001e201fd623323b9044a: Status 404 returned error can't find the container with id 61a96101e1ddf3b54fa2f1a174e5b87ba3e3d4387a6001e201fd623323b9044a Jul 06 23:32:31.793457 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:31.793427 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzxcf" event={"ID":"28f9c1ca-5ec6-4f14-aaba-66dfb0f8fe0c","Type":"ContainerStarted","Data":"61a96101e1ddf3b54fa2f1a174e5b87ba3e3d4387a6001e201fd623323b9044a"} Jul 06 23:32:33.789301 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:33.789270 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w"] Jul 06 23:32:33.792837 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:33.792817 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" Jul 06 23:32:33.796868 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:33.796845 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-js5g9\"" Jul 06 23:32:33.805047 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:33.805022 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzxcf" event={"ID":"28f9c1ca-5ec6-4f14-aaba-66dfb0f8fe0c","Type":"ContainerStarted","Data":"8f4c167a7458f6bf604c0a2c734ad26648af8e1ae9fa8d32cece9bcb98a9b174"} Jul 06 23:32:33.805185 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:33.805169 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzxcf" Jul 06 23:32:33.817960 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:33.817934 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w"] Jul 06 23:32:33.939012 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:33.938961 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzxcf" podStartSLOduration=1.948883399 podStartE2EDuration="3.938942326s" totalImagesPullingTime="1.990058927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:32:33.93723646 +0000 UTC m=+464.651291216" watchObservedRunningTime="2026-07-06 23:32:33.938942326 +0000 UTC m=+464.652996895" Jul 06 23:32:33.948216 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:33.948189 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd5kz\" (UniqueName: \"kubernetes.io/projected/964d482c-6468-4b03-b9c3-dca2177bdd72-kube-api-access-bd5kz\") pod \"limitador-operator-controller-manager-859d7b78bd-lg64w\" (UID: \"964d482c-6468-4b03-b9c3-dca2177bdd72\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" Jul 06 23:32:34.048673 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:34.048588 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd5kz\" (UniqueName: \"kubernetes.io/projected/964d482c-6468-4b03-b9c3-dca2177bdd72-kube-api-access-bd5kz\") pod \"limitador-operator-controller-manager-859d7b78bd-lg64w\" (UID: \"964d482c-6468-4b03-b9c3-dca2177bdd72\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" Jul 06 23:32:34.064395 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:34.064366 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd5kz\" (UniqueName: \"kubernetes.io/projected/964d482c-6468-4b03-b9c3-dca2177bdd72-kube-api-access-bd5kz\") pod \"limitador-operator-controller-manager-859d7b78bd-lg64w\" (UID: \"964d482c-6468-4b03-b9c3-dca2177bdd72\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" Jul 06 23:32:34.103851 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:34.103820 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" Jul 06 23:32:34.256867 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:34.256840 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w"] Jul 06 23:32:34.259697 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:32:34.259669 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964d482c_6468_4b03_b9c3_dca2177bdd72.slice/crio-2ffc332caa0736186be03a72b2c24fd53e969cf457bffa43d42c83d2c4fb762b WatchSource:0}: Error finding container 2ffc332caa0736186be03a72b2c24fd53e969cf457bffa43d42c83d2c4fb762b: Status 404 returned error can't find the container with id 2ffc332caa0736186be03a72b2c24fd53e969cf457bffa43d42c83d2c4fb762b Jul 06 23:32:34.817169 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:34.817129 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" event={"ID":"964d482c-6468-4b03-b9c3-dca2177bdd72","Type":"ContainerStarted","Data":"2ffc332caa0736186be03a72b2c24fd53e969cf457bffa43d42c83d2c4fb762b"} Jul 06 23:32:35.827432 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:35.827401 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" event={"ID":"964d482c-6468-4b03-b9c3-dca2177bdd72","Type":"ContainerStarted","Data":"4ff9f4bfb6711311f1de4af82abd9b42619167f5275256475aa42ba84dfab875"} Jul 06 23:32:35.827815 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:35.827453 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" Jul 06 23:32:35.865352 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:35.865305 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" podStartSLOduration=1.413822649 podStartE2EDuration="2.865289017s" totalImagesPullingTime="1.451466368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:32:35.862611326 +0000 UTC m=+466.576665912" watchObservedRunningTime="2026-07-06 23:32:35.865289017 +0000 UTC m=+466.579343586" Jul 06 23:32:39.110259 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:39.110221 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst"] Jul 06 23:32:39.113046 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:39.113031 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:39.117759 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:39.117707 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-w7r4x\"" Jul 06 23:32:39.133648 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:39.133623 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst"] Jul 06 23:32:39.289838 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:39.289803 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmwdw\" (UniqueName: \"kubernetes.io/projected/40d79115-75d8-46c6-af06-0b1fcd730f9d-kube-api-access-tmwdw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9pmst\" (UID: \"40d79115-75d8-46c6-af06-0b1fcd730f9d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:39.289984 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:39.289858 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40d79115-75d8-46c6-af06-0b1fcd730f9d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9pmst\" (UID: \"40d79115-75d8-46c6-af06-0b1fcd730f9d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:39.391217 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:39.391132 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40d79115-75d8-46c6-af06-0b1fcd730f9d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9pmst\" (UID: \"40d79115-75d8-46c6-af06-0b1fcd730f9d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:39.391217 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:39.391186 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmwdw\" (UniqueName: \"kubernetes.io/projected/40d79115-75d8-46c6-af06-0b1fcd730f9d-kube-api-access-tmwdw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9pmst\" (UID: \"40d79115-75d8-46c6-af06-0b1fcd730f9d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:39.391552 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:39.391534 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40d79115-75d8-46c6-af06-0b1fcd730f9d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9pmst\" (UID: \"40d79115-75d8-46c6-af06-0b1fcd730f9d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:39.408423 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:39.408395 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmwdw\" (UniqueName: \"kubernetes.io/projected/40d79115-75d8-46c6-af06-0b1fcd730f9d-kube-api-access-tmwdw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9pmst\" (UID: \"40d79115-75d8-46c6-af06-0b1fcd730f9d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:39.424274 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:39.424248 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:39.566616 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:39.566586 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst"] Jul 06 23:32:39.567327 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:32:39.567298 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d79115_75d8_46c6_af06_0b1fcd730f9d.slice/crio-c356ce40171c5568d244eb6786bf2b1d09a9b21d78146e58c312f374bed7ab7d WatchSource:0}: Error finding container c356ce40171c5568d244eb6786bf2b1d09a9b21d78146e58c312f374bed7ab7d: Status 404 returned error can't find the container with id c356ce40171c5568d244eb6786bf2b1d09a9b21d78146e58c312f374bed7ab7d Jul 06 23:32:39.847128 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:39.847085 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" event={"ID":"40d79115-75d8-46c6-af06-0b1fcd730f9d","Type":"ContainerStarted","Data":"c356ce40171c5568d244eb6786bf2b1d09a9b21d78146e58c312f374bed7ab7d"} Jul 06 23:32:44.819651 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:44.819624 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzxcf" Jul 06 23:32:44.879428 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:44.878968 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" event={"ID":"40d79115-75d8-46c6-af06-0b1fcd730f9d","Type":"ContainerStarted","Data":"274f6b046ffad97cce74d2e4e26ae86594aee2b79e4a9fd9f8a9afa4d198ec85"} Jul 06 23:32:44.879428 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:44.879164 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:44.901602 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:44.901548 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" podStartSLOduration=1.557708334 podStartE2EDuration="5.90153063s" totalImagesPullingTime="4.343822296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:32:44.89799889 +0000 UTC m=+475.612053464" watchObservedRunningTime="2026-07-06 23:32:44.90153063 +0000 UTC m=+475.615585203" Jul 06 23:32:46.834683 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:46.834658 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" Jul 06 23:32:55.887528 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:55.887499 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:57.526999 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:57.526966 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq"] Jul 06 23:32:57.531971 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:57.531954 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq" Jul 06 23:32:57.547944 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:57.547922 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq"] Jul 06 23:32:57.631316 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:57.631287 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpjxg\" (UniqueName: \"kubernetes.io/projected/9029f253-3cc2-4edf-b714-e78713adc9ec-kube-api-access-bpjxg\") pod \"kuadrant-operator-controller-manager-55467d9f7-cbbsq\" (UID: \"9029f253-3cc2-4edf-b714-e78713adc9ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq" Jul 06 23:32:57.631434 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:57.631364 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9029f253-3cc2-4edf-b714-e78713adc9ec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-cbbsq\" (UID: \"9029f253-3cc2-4edf-b714-e78713adc9ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq" Jul 06 23:32:57.731750 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:57.731711 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpjxg\" (UniqueName: \"kubernetes.io/projected/9029f253-3cc2-4edf-b714-e78713adc9ec-kube-api-access-bpjxg\") pod \"kuadrant-operator-controller-manager-55467d9f7-cbbsq\" (UID: \"9029f253-3cc2-4edf-b714-e78713adc9ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq" Jul 06 23:32:57.731854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:57.731783 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9029f253-3cc2-4edf-b714-e78713adc9ec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-cbbsq\" (UID: \"9029f253-3cc2-4edf-b714-e78713adc9ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq" Jul 06 23:32:57.732066 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:57.732052 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9029f253-3cc2-4edf-b714-e78713adc9ec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-cbbsq\" (UID: \"9029f253-3cc2-4edf-b714-e78713adc9ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq" Jul 06 23:32:57.742372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:57.742346 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpjxg\" (UniqueName: \"kubernetes.io/projected/9029f253-3cc2-4edf-b714-e78713adc9ec-kube-api-access-bpjxg\") pod \"kuadrant-operator-controller-manager-55467d9f7-cbbsq\" (UID: \"9029f253-3cc2-4edf-b714-e78713adc9ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq" Jul 06 23:32:57.841699 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:57.841648 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq" Jul 06 23:32:57.971372 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:57.971350 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq"] Jul 06 23:32:57.974029 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:32:57.974006 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9029f253_3cc2_4edf_b714_e78713adc9ec.slice/crio-d718cc5b30039f6e753a705ecc51f683153928feaaadfa4b2d3c539d449ac0b4 WatchSource:0}: Error finding container d718cc5b30039f6e753a705ecc51f683153928feaaadfa4b2d3c539d449ac0b4: Status 404 returned error can't find the container with id d718cc5b30039f6e753a705ecc51f683153928feaaadfa4b2d3c539d449ac0b4 Jul 06 23:32:57.979268 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:57.979248 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq"] Jul 06 23:32:57.985733 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:57.985667 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq"] Jul 06 23:32:58.001655 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.001633 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst"] Jul 06 23:32:58.001886 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.001836 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" podUID="40d79115-75d8-46c6-af06-0b1fcd730f9d" containerName="manager" containerID="cri-o://274f6b046ffad97cce74d2e4e26ae86594aee2b79e4a9fd9f8a9afa4d198ec85" gracePeriod=2 Jul 06 23:32:58.006052 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.006031 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn"] Jul 06 23:32:58.011793 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.011711 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn" Jul 06 23:32:58.013909 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.013890 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst"] Jul 06 23:32:58.021099 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.021079 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn"] Jul 06 23:32:58.024167 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:58.023969 2571 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"40d79115-75d8-46c6-af06-0b1fcd730f9d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-9pmst\": pods \"kuadrant-operator-controller-manager-8684c9ff49-9pmst\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:58.025320 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.025297 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4"] Jul 06 23:32:58.025708 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.025687 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d79115-75d8-46c6-af06-0b1fcd730f9d" containerName="manager" Jul 06 23:32:58.025801 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.025711 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d79115-75d8-46c6-af06-0b1fcd730f9d" containerName="manager" Jul 06 23:32:58.025865 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.025826 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="40d79115-75d8-46c6-af06-0b1fcd730f9d" containerName="manager" Jul 06 23:32:58.027046 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:58.027022 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9pmst\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="40d79115-75d8-46c6-af06-0b1fcd730f9d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:58.030491 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.030473 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4" Jul 06 23:32:58.032839 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.032822 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w"] Jul 06 23:32:58.032994 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.032977 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" podUID="964d482c-6468-4b03-b9c3-dca2177bdd72" containerName="manager" containerID="cri-o://4ff9f4bfb6711311f1de4af82abd9b42619167f5275256475aa42ba84dfab875" gracePeriod=2 Jul 06 23:32:58.033667 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.033645 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70749cf6-bb66-4d01-b0dc-a06f3ec7ce78-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-55rfn\" (UID: \"70749cf6-bb66-4d01-b0dc-a06f3ec7ce78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn" Jul 06 23:32:58.033761 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.033735 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j8mp\" (UniqueName: \"kubernetes.io/projected/70749cf6-bb66-4d01-b0dc-a06f3ec7ce78-kube-api-access-5j8mp\") pod \"kuadrant-operator-controller-manager-55467d9f7-55rfn\" (UID: \"70749cf6-bb66-4d01-b0dc-a06f3ec7ce78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn" Jul 06 23:32:58.046916 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.046884 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4"] Jul 06 23:32:58.049497 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.049475 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w"] Jul 06 23:32:58.062067 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:58.062038 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9pmst\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="40d79115-75d8-46c6-af06-0b1fcd730f9d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:58.062253 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.062238 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t44rq"] Jul 06 23:32:58.062505 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.062495 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="964d482c-6468-4b03-b9c3-dca2177bdd72" containerName="manager" Jul 06 23:32:58.062538 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.062507 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="964d482c-6468-4b03-b9c3-dca2177bdd72" containerName="manager" Jul 06 23:32:58.062569 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.062564 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="964d482c-6468-4b03-b9c3-dca2177bdd72" containerName="manager" Jul 06 23:32:58.066305 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.066287 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t44rq" Jul 06 23:32:58.074291 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.074271 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t44rq"] Jul 06 23:32:58.081416 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:58.081391 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lg64w\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="964d482c-6468-4b03-b9c3-dca2177bdd72" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" Jul 06 23:32:58.107946 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:58.107883 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9pmst\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="40d79115-75d8-46c6-af06-0b1fcd730f9d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:58.109832 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:58.109810 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lg64w\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="964d482c-6468-4b03-b9c3-dca2177bdd72" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" Jul 06 23:32:58.134104 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.134078 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smxph\" (UniqueName: \"kubernetes.io/projected/ebe5f1bf-a1cc-4a28-a938-9bc1a11c4e4e-kube-api-access-smxph\") pod \"limitador-operator-controller-manager-859d7b78bd-t44rq\" (UID: \"ebe5f1bf-a1cc-4a28-a938-9bc1a11c4e4e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t44rq" Jul 06 23:32:58.134228 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.134147 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70749cf6-bb66-4d01-b0dc-a06f3ec7ce78-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-55rfn\" (UID: \"70749cf6-bb66-4d01-b0dc-a06f3ec7ce78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn" Jul 06 23:32:58.134387 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.134237 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmpz4\" (UniqueName: \"kubernetes.io/projected/7f7ff9cd-7608-4013-b759-357664099d6b-kube-api-access-bmpz4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-w5qv4\" (UID: \"7f7ff9cd-7608-4013-b759-357664099d6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4" Jul 06 23:32:58.134387 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.134276 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j8mp\" (UniqueName: \"kubernetes.io/projected/70749cf6-bb66-4d01-b0dc-a06f3ec7ce78-kube-api-access-5j8mp\") pod \"kuadrant-operator-controller-manager-55467d9f7-55rfn\" (UID: \"70749cf6-bb66-4d01-b0dc-a06f3ec7ce78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn" Jul 06 23:32:58.134387 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.134334 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f7ff9cd-7608-4013-b759-357664099d6b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-w5qv4\" (UID: \"7f7ff9cd-7608-4013-b759-357664099d6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4" Jul 06 23:32:58.134542 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.134483 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70749cf6-bb66-4d01-b0dc-a06f3ec7ce78-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-55rfn\" (UID: \"70749cf6-bb66-4d01-b0dc-a06f3ec7ce78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn" Jul 06 23:32:58.144811 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.144789 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j8mp\" (UniqueName: \"kubernetes.io/projected/70749cf6-bb66-4d01-b0dc-a06f3ec7ce78-kube-api-access-5j8mp\") pod \"kuadrant-operator-controller-manager-55467d9f7-55rfn\" (UID: \"70749cf6-bb66-4d01-b0dc-a06f3ec7ce78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn" Jul 06 23:32:58.235373 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.235345 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmpz4\" (UniqueName: \"kubernetes.io/projected/7f7ff9cd-7608-4013-b759-357664099d6b-kube-api-access-bmpz4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-w5qv4\" (UID: \"7f7ff9cd-7608-4013-b759-357664099d6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4" Jul 06 23:32:58.235503 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.235413 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f7ff9cd-7608-4013-b759-357664099d6b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-w5qv4\" (UID: \"7f7ff9cd-7608-4013-b759-357664099d6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4" Jul 06 23:32:58.235503 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.235455 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smxph\" (UniqueName: \"kubernetes.io/projected/ebe5f1bf-a1cc-4a28-a938-9bc1a11c4e4e-kube-api-access-smxph\") pod \"limitador-operator-controller-manager-859d7b78bd-t44rq\" (UID: \"ebe5f1bf-a1cc-4a28-a938-9bc1a11c4e4e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t44rq" Jul 06 23:32:58.235842 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.235821 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f7ff9cd-7608-4013-b759-357664099d6b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-w5qv4\" (UID: \"7f7ff9cd-7608-4013-b759-357664099d6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4" Jul 06 23:32:58.246848 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.246821 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmpz4\" (UniqueName: \"kubernetes.io/projected/7f7ff9cd-7608-4013-b759-357664099d6b-kube-api-access-bmpz4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-w5qv4\" (UID: \"7f7ff9cd-7608-4013-b759-357664099d6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4" Jul 06 23:32:58.247174 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.247134 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smxph\" (UniqueName: \"kubernetes.io/projected/ebe5f1bf-a1cc-4a28-a938-9bc1a11c4e4e-kube-api-access-smxph\") pod \"limitador-operator-controller-manager-859d7b78bd-t44rq\" (UID: \"ebe5f1bf-a1cc-4a28-a938-9bc1a11c4e4e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t44rq" Jul 06 23:32:58.257796 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.257777 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:58.259847 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:58.259824 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9pmst\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="40d79115-75d8-46c6-af06-0b1fcd730f9d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:58.261330 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.261314 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" Jul 06 23:32:58.261458 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:58.261437 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lg64w\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="964d482c-6468-4b03-b9c3-dca2177bdd72" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" Jul 06 23:32:58.263164 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:58.263146 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9pmst\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="40d79115-75d8-46c6-af06-0b1fcd730f9d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:58.264687 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:58.264668 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lg64w\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="964d482c-6468-4b03-b9c3-dca2177bdd72" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" Jul 06 23:32:58.335760 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.335731 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd5kz\" (UniqueName: \"kubernetes.io/projected/964d482c-6468-4b03-b9c3-dca2177bdd72-kube-api-access-bd5kz\") pod \"964d482c-6468-4b03-b9c3-dca2177bdd72\" (UID: \"964d482c-6468-4b03-b9c3-dca2177bdd72\") " Jul 06 23:32:58.335846 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.335808 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40d79115-75d8-46c6-af06-0b1fcd730f9d-extensions-socket-volume\") pod \"40d79115-75d8-46c6-af06-0b1fcd730f9d\" (UID: \"40d79115-75d8-46c6-af06-0b1fcd730f9d\") " Jul 06 23:32:58.335886 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.335860 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmwdw\" (UniqueName: \"kubernetes.io/projected/40d79115-75d8-46c6-af06-0b1fcd730f9d-kube-api-access-tmwdw\") pod \"40d79115-75d8-46c6-af06-0b1fcd730f9d\" (UID: \"40d79115-75d8-46c6-af06-0b1fcd730f9d\") " Jul 06 23:32:58.336206 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.336182 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d79115-75d8-46c6-af06-0b1fcd730f9d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "40d79115-75d8-46c6-af06-0b1fcd730f9d" (UID: "40d79115-75d8-46c6-af06-0b1fcd730f9d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:32:58.338376 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.338358 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964d482c-6468-4b03-b9c3-dca2177bdd72-kube-api-access-bd5kz" (OuterVolumeSpecName: "kube-api-access-bd5kz") pod "964d482c-6468-4b03-b9c3-dca2177bdd72" (UID: "964d482c-6468-4b03-b9c3-dca2177bdd72"). InnerVolumeSpecName "kube-api-access-bd5kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:32:58.338376 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.338363 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d79115-75d8-46c6-af06-0b1fcd730f9d-kube-api-access-tmwdw" (OuterVolumeSpecName: "kube-api-access-tmwdw") pod "40d79115-75d8-46c6-af06-0b1fcd730f9d" (UID: "40d79115-75d8-46c6-af06-0b1fcd730f9d"). InnerVolumeSpecName "kube-api-access-tmwdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:32:58.431123 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.431100 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn" Jul 06 23:32:58.436900 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.436880 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40d79115-75d8-46c6-af06-0b1fcd730f9d-extensions-socket-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:32:58.436948 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.436903 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmwdw\" (UniqueName: \"kubernetes.io/projected/40d79115-75d8-46c6-af06-0b1fcd730f9d-kube-api-access-tmwdw\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:32:58.436948 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.436914 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd5kz\" (UniqueName: \"kubernetes.io/projected/964d482c-6468-4b03-b9c3-dca2177bdd72-kube-api-access-bd5kz\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:32:58.438882 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.438869 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4" Jul 06 23:32:58.445690 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.445675 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t44rq" Jul 06 23:32:58.597259 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.597228 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn"] Jul 06 23:32:58.600213 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:32:58.600167 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70749cf6_bb66_4d01_b0dc_a06f3ec7ce78.slice/crio-a15174a33766e8f7c24231595baa3fbbb12bd884199f2d733897c09aa0e54bf7 WatchSource:0}: Error finding container a15174a33766e8f7c24231595baa3fbbb12bd884199f2d733897c09aa0e54bf7: Status 404 returned error can't find the container with id a15174a33766e8f7c24231595baa3fbbb12bd884199f2d733897c09aa0e54bf7 Jul 06 23:32:58.614989 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.614965 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4"] Jul 06 23:32:58.618592 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:32:58.618523 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7ff9cd_7608_4013_b759_357664099d6b.slice/crio-562823cf6fccfd62e3c394555875575feb4982f561c547d392006a08487533ae WatchSource:0}: Error finding container 562823cf6fccfd62e3c394555875575feb4982f561c547d392006a08487533ae: Status 404 returned error can't find the container with id 562823cf6fccfd62e3c394555875575feb4982f561c547d392006a08487533ae Jul 06 23:32:58.630515 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.630499 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t44rq"] Jul 06 23:32:58.635839 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:32:58.635810 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe5f1bf_a1cc_4a28_a938_9bc1a11c4e4e.slice/crio-d46141587e41f4cae070945edb7cc6db925b64456ca1710315dd14831e74612f WatchSource:0}: Error finding container d46141587e41f4cae070945edb7cc6db925b64456ca1710315dd14831e74612f: Status 404 returned error can't find the container with id d46141587e41f4cae070945edb7cc6db925b64456ca1710315dd14831e74612f Jul 06 23:32:58.946753 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.946643 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn" event={"ID":"70749cf6-bb66-4d01-b0dc-a06f3ec7ce78","Type":"ContainerStarted","Data":"0919a641ad9c0b643333bc5aaefd1b7f64d11d105f36bdcac3612e5db99d032c"} Jul 06 23:32:58.946919 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.946764 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn" Jul 06 23:32:58.946919 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.946788 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn" event={"ID":"70749cf6-bb66-4d01-b0dc-a06f3ec7ce78","Type":"ContainerStarted","Data":"a15174a33766e8f7c24231595baa3fbbb12bd884199f2d733897c09aa0e54bf7"} Jul 06 23:32:58.947947 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.947923 2571 generic.go:356] "Generic (PLEG): container finished" podID="964d482c-6468-4b03-b9c3-dca2177bdd72" containerID="4ff9f4bfb6711311f1de4af82abd9b42619167f5275256475aa42ba84dfab875" exitCode=0 Jul 06 23:32:58.948034 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.947979 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" Jul 06 23:32:58.948034 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.947997 2571 scope.go:117] "RemoveContainer" containerID="4ff9f4bfb6711311f1de4af82abd9b42619167f5275256475aa42ba84dfab875" Jul 06 23:32:58.949868 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:58.949840 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lg64w\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="964d482c-6468-4b03-b9c3-dca2177bdd72" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" Jul 06 23:32:58.950003 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.949839 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq" podUID="9029f253-3cc2-4edf-b714-e78713adc9ec" containerName="manager" containerID="cri-o://c3b67f88d84466b34e027a6a45ec9bc96b2fe66a0c2876a60cfa61fcee6cc3c8" gracePeriod=2 Jul 06 23:32:58.951610 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.951587 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4" event={"ID":"7f7ff9cd-7608-4013-b759-357664099d6b","Type":"ContainerStarted","Data":"1f6974e5f080870032c8d3a665f9159b3fc64897e4c45492d8aa18af4bca72b0"} Jul 06 23:32:58.951711 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.951633 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4" event={"ID":"7f7ff9cd-7608-4013-b759-357664099d6b","Type":"ContainerStarted","Data":"562823cf6fccfd62e3c394555875575feb4982f561c547d392006a08487533ae"} Jul 06 23:32:58.951711 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:58.951708 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9pmst\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="40d79115-75d8-46c6-af06-0b1fcd730f9d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:58.951870 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.951731 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4" Jul 06 23:32:58.953049 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.953015 2571 generic.go:356] "Generic (PLEG): container finished" podID="40d79115-75d8-46c6-af06-0b1fcd730f9d" containerID="274f6b046ffad97cce74d2e4e26ae86594aee2b79e4a9fd9f8a9afa4d198ec85" exitCode=0 Jul 06 23:32:58.953131 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.953069 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:58.955605 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.955570 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t44rq" event={"ID":"ebe5f1bf-a1cc-4a28-a938-9bc1a11c4e4e","Type":"ContainerStarted","Data":"5c109dde51446e277456a11f0cf0ec6b48249bfccb782eb0d1d301101f6daaf8"} Jul 06 23:32:58.955658 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.955619 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t44rq" event={"ID":"ebe5f1bf-a1cc-4a28-a938-9bc1a11c4e4e","Type":"ContainerStarted","Data":"d46141587e41f4cae070945edb7cc6db925b64456ca1710315dd14831e74612f"} Jul 06 23:32:58.955706 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.955682 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t44rq" Jul 06 23:32:58.958948 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.958931 2571 scope.go:117] "RemoveContainer" containerID="4ff9f4bfb6711311f1de4af82abd9b42619167f5275256475aa42ba84dfab875" Jul 06 23:32:58.959220 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:58.959201 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ff9f4bfb6711311f1de4af82abd9b42619167f5275256475aa42ba84dfab875\": container with ID starting with 4ff9f4bfb6711311f1de4af82abd9b42619167f5275256475aa42ba84dfab875 not found: ID does not exist" containerID="4ff9f4bfb6711311f1de4af82abd9b42619167f5275256475aa42ba84dfab875" Jul 06 23:32:58.959291 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.959227 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ff9f4bfb6711311f1de4af82abd9b42619167f5275256475aa42ba84dfab875"} err="failed to get container status \"4ff9f4bfb6711311f1de4af82abd9b42619167f5275256475aa42ba84dfab875\": rpc error: code = NotFound desc = could not find container \"4ff9f4bfb6711311f1de4af82abd9b42619167f5275256475aa42ba84dfab875\": container with ID starting with 4ff9f4bfb6711311f1de4af82abd9b42619167f5275256475aa42ba84dfab875 not found: ID does not exist" Jul 06 23:32:58.959291 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.959243 2571 scope.go:117] "RemoveContainer" containerID="274f6b046ffad97cce74d2e4e26ae86594aee2b79e4a9fd9f8a9afa4d198ec85" Jul 06 23:32:58.979782 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.979742 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn" podStartSLOduration=1.9797053949999999 podStartE2EDuration="1.979705395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:32:58.977128089 +0000 UTC m=+489.691182660" watchObservedRunningTime="2026-07-06 23:32:58.979705395 +0000 UTC m=+489.693759964" Jul 06 23:32:58.994528 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:58.994488 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t44rq" podStartSLOduration=0.994476029 podStartE2EDuration="994.476029ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:32:58.992920051 +0000 UTC m=+489.706974624" watchObservedRunningTime="2026-07-06 23:32:58.994476029 +0000 UTC m=+489.708530666" Jul 06 23:32:58.994653 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:58.994630 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9pmst\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="40d79115-75d8-46c6-af06-0b1fcd730f9d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:58.996389 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:58.996369 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-cbbsq\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="9029f253-3cc2-4edf-b714-e78713adc9ec" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq" Jul 06 23:32:59.016422 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:59.016393 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lg64w\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="964d482c-6468-4b03-b9c3-dca2177bdd72" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" Jul 06 23:32:59.017731 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:59.017671 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4" podStartSLOduration=1.017652108 podStartE2EDuration="1.017652108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:32:59.014475998 +0000 UTC m=+489.728530569" watchObservedRunningTime="2026-07-06 23:32:59.017652108 +0000 UTC m=+489.731706677" Jul 06 23:32:59.018050 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:59.018032 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9pmst\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="40d79115-75d8-46c6-af06-0b1fcd730f9d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:59.019645 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:59.019624 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-cbbsq\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="9029f253-3cc2-4edf-b714-e78713adc9ec" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq" Jul 06 23:32:59.023275 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:59.023257 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lg64w\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="964d482c-6468-4b03-b9c3-dca2177bdd72" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" Jul 06 23:32:59.061926 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:59.061905 2571 scope.go:117] "RemoveContainer" containerID="274f6b046ffad97cce74d2e4e26ae86594aee2b79e4a9fd9f8a9afa4d198ec85" Jul 06 23:32:59.062216 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:59.062199 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"274f6b046ffad97cce74d2e4e26ae86594aee2b79e4a9fd9f8a9afa4d198ec85\": container with ID starting with 274f6b046ffad97cce74d2e4e26ae86594aee2b79e4a9fd9f8a9afa4d198ec85 not found: ID does not exist" containerID="274f6b046ffad97cce74d2e4e26ae86594aee2b79e4a9fd9f8a9afa4d198ec85" Jul 06 23:32:59.062265 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:59.062223 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"274f6b046ffad97cce74d2e4e26ae86594aee2b79e4a9fd9f8a9afa4d198ec85"} err="failed to get container status \"274f6b046ffad97cce74d2e4e26ae86594aee2b79e4a9fd9f8a9afa4d198ec85\": rpc error: code = NotFound desc = could not find container \"274f6b046ffad97cce74d2e4e26ae86594aee2b79e4a9fd9f8a9afa4d198ec85\": container with ID starting with 274f6b046ffad97cce74d2e4e26ae86594aee2b79e4a9fd9f8a9afa4d198ec85 not found: ID does not exist" Jul 06 23:32:59.193517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:59.193495 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq" Jul 06 23:32:59.195684 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:59.195660 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lg64w\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="964d482c-6468-4b03-b9c3-dca2177bdd72" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" Jul 06 23:32:59.197477 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:59.197429 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9pmst\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="40d79115-75d8-46c6-af06-0b1fcd730f9d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:59.199374 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:59.199348 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-cbbsq\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="9029f253-3cc2-4edf-b714-e78713adc9ec" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq" Jul 06 23:32:59.243707 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:59.243681 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9029f253-3cc2-4edf-b714-e78713adc9ec-extensions-socket-volume\") pod \"9029f253-3cc2-4edf-b714-e78713adc9ec\" (UID: \"9029f253-3cc2-4edf-b714-e78713adc9ec\") " Jul 06 23:32:59.243843 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:59.243769 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpjxg\" (UniqueName: \"kubernetes.io/projected/9029f253-3cc2-4edf-b714-e78713adc9ec-kube-api-access-bpjxg\") pod \"9029f253-3cc2-4edf-b714-e78713adc9ec\" (UID: \"9029f253-3cc2-4edf-b714-e78713adc9ec\") " Jul 06 23:32:59.243941 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:59.243921 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9029f253-3cc2-4edf-b714-e78713adc9ec-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9029f253-3cc2-4edf-b714-e78713adc9ec" (UID: "9029f253-3cc2-4edf-b714-e78713adc9ec"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:32:59.245880 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:59.245849 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9029f253-3cc2-4edf-b714-e78713adc9ec-kube-api-access-bpjxg" (OuterVolumeSpecName: "kube-api-access-bpjxg") pod "9029f253-3cc2-4edf-b714-e78713adc9ec" (UID: "9029f253-3cc2-4edf-b714-e78713adc9ec"). InnerVolumeSpecName "kube-api-access-bpjxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:32:59.345274 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:59.345242 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpjxg\" (UniqueName: \"kubernetes.io/projected/9029f253-3cc2-4edf-b714-e78713adc9ec-kube-api-access-bpjxg\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:32:59.345274 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:59.345274 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9029f253-3cc2-4edf-b714-e78713adc9ec-extensions-socket-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:32:59.784420 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:59.784389 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9pmst\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="40d79115-75d8-46c6-af06-0b1fcd730f9d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9pmst" Jul 06 23:32:59.785471 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:59.785451 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40d79115-75d8-46c6-af06-0b1fcd730f9d" path="/var/lib/kubelet/pods/40d79115-75d8-46c6-af06-0b1fcd730f9d/volumes" Jul 06 23:32:59.785844 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:59.785830 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9029f253-3cc2-4edf-b714-e78713adc9ec" path="/var/lib/kubelet/pods/9029f253-3cc2-4edf-b714-e78713adc9ec/volumes" Jul 06 23:32:59.786136 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:59.786122 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="964d482c-6468-4b03-b9c3-dca2177bdd72" path="/var/lib/kubelet/pods/964d482c-6468-4b03-b9c3-dca2177bdd72/volumes" Jul 06 23:32:59.786227 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:59.786210 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-cbbsq\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="9029f253-3cc2-4edf-b714-e78713adc9ec" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq" Jul 06 23:32:59.787955 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:59.787938 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lg64w\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="964d482c-6468-4b03-b9c3-dca2177bdd72" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lg64w" Jul 06 23:32:59.964126 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:59.964097 2571 generic.go:356] "Generic (PLEG): container finished" podID="9029f253-3cc2-4edf-b714-e78713adc9ec" containerID="c3b67f88d84466b34e027a6a45ec9bc96b2fe66a0c2876a60cfa61fcee6cc3c8" exitCode=2 Jul 06 23:32:59.964298 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:59.964137 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq" Jul 06 23:32:59.964298 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:59.964186 2571 scope.go:117] "RemoveContainer" containerID="c3b67f88d84466b34e027a6a45ec9bc96b2fe66a0c2876a60cfa61fcee6cc3c8" Jul 06 23:32:59.968487 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:59.968445 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-cbbsq\" is forbidden: User \"system:node:ip-10-0-135-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-248.ec2.internal' and this object" podUID="9029f253-3cc2-4edf-b714-e78713adc9ec" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-cbbsq" Jul 06 23:32:59.972902 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:59.972888 2571 scope.go:117] "RemoveContainer" containerID="c3b67f88d84466b34e027a6a45ec9bc96b2fe66a0c2876a60cfa61fcee6cc3c8" Jul 06 23:32:59.973154 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:32:59.973137 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3b67f88d84466b34e027a6a45ec9bc96b2fe66a0c2876a60cfa61fcee6cc3c8\": container with ID starting with c3b67f88d84466b34e027a6a45ec9bc96b2fe66a0c2876a60cfa61fcee6cc3c8 not found: ID does not exist" containerID="c3b67f88d84466b34e027a6a45ec9bc96b2fe66a0c2876a60cfa61fcee6cc3c8" Jul 06 23:32:59.973199 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:32:59.973164 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3b67f88d84466b34e027a6a45ec9bc96b2fe66a0c2876a60cfa61fcee6cc3c8"} err="failed to get container status \"c3b67f88d84466b34e027a6a45ec9bc96b2fe66a0c2876a60cfa61fcee6cc3c8\": rpc error: code = NotFound desc = could not find container \"c3b67f88d84466b34e027a6a45ec9bc96b2fe66a0c2876a60cfa61fcee6cc3c8\": container with ID starting with c3b67f88d84466b34e027a6a45ec9bc96b2fe66a0c2876a60cfa61fcee6cc3c8 not found: ID does not exist" Jul 06 23:33:09.966638 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:09.966611 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4" Jul 06 23:33:09.967188 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:09.966793 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn" Jul 06 23:33:09.967188 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:09.966861 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t44rq" Jul 06 23:33:10.056947 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.056917 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4"] Jul 06 23:33:10.057142 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.057119 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4" podUID="7f7ff9cd-7608-4013-b759-357664099d6b" containerName="manager" containerID="cri-o://1f6974e5f080870032c8d3a665f9159b3fc64897e4c45492d8aa18af4bca72b0" gracePeriod=10 Jul 06 23:33:10.297289 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.297264 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4" Jul 06 23:33:10.325282 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.325259 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f7ff9cd-7608-4013-b759-357664099d6b-extensions-socket-volume\") pod \"7f7ff9cd-7608-4013-b759-357664099d6b\" (UID: \"7f7ff9cd-7608-4013-b759-357664099d6b\") " Jul 06 23:33:10.325375 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.325288 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmpz4\" (UniqueName: \"kubernetes.io/projected/7f7ff9cd-7608-4013-b759-357664099d6b-kube-api-access-bmpz4\") pod \"7f7ff9cd-7608-4013-b759-357664099d6b\" (UID: \"7f7ff9cd-7608-4013-b759-357664099d6b\") " Jul 06 23:33:10.325594 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.325556 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7ff9cd-7608-4013-b759-357664099d6b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7f7ff9cd-7608-4013-b759-357664099d6b" (UID: "7f7ff9cd-7608-4013-b759-357664099d6b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:33:10.327354 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.327332 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7ff9cd-7608-4013-b759-357664099d6b-kube-api-access-bmpz4" (OuterVolumeSpecName: "kube-api-access-bmpz4") pod "7f7ff9cd-7608-4013-b759-357664099d6b" (UID: "7f7ff9cd-7608-4013-b759-357664099d6b"). InnerVolumeSpecName "kube-api-access-bmpz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:33:10.367593 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.367569 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b"] Jul 06 23:33:10.367943 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.367918 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7ff9cd-7608-4013-b759-357664099d6b" containerName="manager" Jul 06 23:33:10.367943 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.367946 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7ff9cd-7608-4013-b759-357664099d6b" containerName="manager" Jul 06 23:33:10.368084 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.367968 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9029f253-3cc2-4edf-b714-e78713adc9ec" containerName="manager" Jul 06 23:33:10.368084 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.367976 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9029f253-3cc2-4edf-b714-e78713adc9ec" containerName="manager" Jul 06 23:33:10.368084 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.368072 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9029f253-3cc2-4edf-b714-e78713adc9ec" containerName="manager" Jul 06 23:33:10.368084 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.368085 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f7ff9cd-7608-4013-b759-357664099d6b" containerName="manager" Jul 06 23:33:10.369997 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.369980 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b" Jul 06 23:33:10.394160 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.394133 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b"] Jul 06 23:33:10.425830 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.425795 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7955cb15-e2d6-4a22-8013-e97bb65c831d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jgq9b\" (UID: \"7955cb15-e2d6-4a22-8013-e97bb65c831d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b" Jul 06 23:33:10.425966 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.425844 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cvxg\" (UniqueName: \"kubernetes.io/projected/7955cb15-e2d6-4a22-8013-e97bb65c831d-kube-api-access-4cvxg\") pod \"kuadrant-operator-controller-manager-86878c769f-jgq9b\" (UID: \"7955cb15-e2d6-4a22-8013-e97bb65c831d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b" Jul 06 23:33:10.425966 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.425940 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f7ff9cd-7608-4013-b759-357664099d6b-extensions-socket-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:33:10.425966 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.425957 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmpz4\" (UniqueName: \"kubernetes.io/projected/7f7ff9cd-7608-4013-b759-357664099d6b-kube-api-access-bmpz4\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:33:10.526456 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.526367 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7955cb15-e2d6-4a22-8013-e97bb65c831d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jgq9b\" (UID: \"7955cb15-e2d6-4a22-8013-e97bb65c831d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b" Jul 06 23:33:10.526456 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.526417 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cvxg\" (UniqueName: \"kubernetes.io/projected/7955cb15-e2d6-4a22-8013-e97bb65c831d-kube-api-access-4cvxg\") pod \"kuadrant-operator-controller-manager-86878c769f-jgq9b\" (UID: \"7955cb15-e2d6-4a22-8013-e97bb65c831d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b" Jul 06 23:33:10.526807 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.526790 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7955cb15-e2d6-4a22-8013-e97bb65c831d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jgq9b\" (UID: \"7955cb15-e2d6-4a22-8013-e97bb65c831d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b" Jul 06 23:33:10.537199 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.537176 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cvxg\" (UniqueName: \"kubernetes.io/projected/7955cb15-e2d6-4a22-8013-e97bb65c831d-kube-api-access-4cvxg\") pod \"kuadrant-operator-controller-manager-86878c769f-jgq9b\" (UID: \"7955cb15-e2d6-4a22-8013-e97bb65c831d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b" Jul 06 23:33:10.678798 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.678768 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b" Jul 06 23:33:10.811547 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:10.811523 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b"] Jul 06 23:33:10.813970 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:33:10.813947 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7955cb15_e2d6_4a22_8013_e97bb65c831d.slice/crio-66f18783ceede868d0b1893d5388cdf05342f66d4e454ad4dd1e812ac725f3fa WatchSource:0}: Error finding container 66f18783ceede868d0b1893d5388cdf05342f66d4e454ad4dd1e812ac725f3fa: Status 404 returned error can't find the container with id 66f18783ceede868d0b1893d5388cdf05342f66d4e454ad4dd1e812ac725f3fa Jul 06 23:33:11.020922 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:11.020888 2571 generic.go:356] "Generic (PLEG): container finished" podID="7f7ff9cd-7608-4013-b759-357664099d6b" containerID="1f6974e5f080870032c8d3a665f9159b3fc64897e4c45492d8aa18af4bca72b0" exitCode=0 Jul 06 23:33:11.021364 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:11.020944 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4" Jul 06 23:33:11.021364 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:11.020973 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4" event={"ID":"7f7ff9cd-7608-4013-b759-357664099d6b","Type":"ContainerDied","Data":"1f6974e5f080870032c8d3a665f9159b3fc64897e4c45492d8aa18af4bca72b0"} Jul 06 23:33:11.021364 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:11.021069 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4" event={"ID":"7f7ff9cd-7608-4013-b759-357664099d6b","Type":"ContainerDied","Data":"562823cf6fccfd62e3c394555875575feb4982f561c547d392006a08487533ae"} Jul 06 23:33:11.021364 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:11.021102 2571 scope.go:117] "RemoveContainer" containerID="1f6974e5f080870032c8d3a665f9159b3fc64897e4c45492d8aa18af4bca72b0" Jul 06 23:33:11.022770 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:11.022742 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b" event={"ID":"7955cb15-e2d6-4a22-8013-e97bb65c831d","Type":"ContainerStarted","Data":"779e6a01aba1792748ff14127adb2bb7aa98d93db853846b50a6a04dfd5bc228"} Jul 06 23:33:11.022877 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:11.022795 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b" event={"ID":"7955cb15-e2d6-4a22-8013-e97bb65c831d","Type":"ContainerStarted","Data":"66f18783ceede868d0b1893d5388cdf05342f66d4e454ad4dd1e812ac725f3fa"} Jul 06 23:33:11.022877 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:11.022853 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b" Jul 06 23:33:11.030489 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:11.030473 2571 scope.go:117] "RemoveContainer" containerID="1f6974e5f080870032c8d3a665f9159b3fc64897e4c45492d8aa18af4bca72b0" Jul 06 23:33:11.030756 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:33:11.030736 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6974e5f080870032c8d3a665f9159b3fc64897e4c45492d8aa18af4bca72b0\": container with ID starting with 1f6974e5f080870032c8d3a665f9159b3fc64897e4c45492d8aa18af4bca72b0 not found: ID does not exist" containerID="1f6974e5f080870032c8d3a665f9159b3fc64897e4c45492d8aa18af4bca72b0" Jul 06 23:33:11.030815 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:11.030768 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6974e5f080870032c8d3a665f9159b3fc64897e4c45492d8aa18af4bca72b0"} err="failed to get container status \"1f6974e5f080870032c8d3a665f9159b3fc64897e4c45492d8aa18af4bca72b0\": rpc error: code = NotFound desc = could not find container \"1f6974e5f080870032c8d3a665f9159b3fc64897e4c45492d8aa18af4bca72b0\": container with ID starting with 1f6974e5f080870032c8d3a665f9159b3fc64897e4c45492d8aa18af4bca72b0 not found: ID does not exist" Jul 06 23:33:11.058879 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:11.058849 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4"] Jul 06 23:33:11.066580 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:11.066524 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5qv4"] Jul 06 23:33:11.095587 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:11.095542 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b" podStartSLOduration=1.095526759 podStartE2EDuration="1.095526759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:33:11.09516861 +0000 UTC m=+501.809223189" watchObservedRunningTime="2026-07-06 23:33:11.095526759 +0000 UTC m=+501.809581328" Jul 06 23:33:11.785536 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:11.785504 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f7ff9cd-7608-4013-b759-357664099d6b" path="/var/lib/kubelet/pods/7f7ff9cd-7608-4013-b759-357664099d6b/volumes" Jul 06 23:33:22.031504 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:22.031468 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b" Jul 06 23:33:22.079831 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:22.079802 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn"] Jul 06 23:33:22.080037 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:22.080011 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn" podUID="70749cf6-bb66-4d01-b0dc-a06f3ec7ce78" containerName="manager" containerID="cri-o://0919a641ad9c0b643333bc5aaefd1b7f64d11d105f36bdcac3612e5db99d032c" gracePeriod=10 Jul 06 23:33:22.323190 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:22.323161 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn" Jul 06 23:33:22.411534 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:22.411502 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j8mp\" (UniqueName: \"kubernetes.io/projected/70749cf6-bb66-4d01-b0dc-a06f3ec7ce78-kube-api-access-5j8mp\") pod \"70749cf6-bb66-4d01-b0dc-a06f3ec7ce78\" (UID: \"70749cf6-bb66-4d01-b0dc-a06f3ec7ce78\") " Jul 06 23:33:22.411708 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:22.411595 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70749cf6-bb66-4d01-b0dc-a06f3ec7ce78-extensions-socket-volume\") pod \"70749cf6-bb66-4d01-b0dc-a06f3ec7ce78\" (UID: \"70749cf6-bb66-4d01-b0dc-a06f3ec7ce78\") " Jul 06 23:33:22.412022 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:22.412001 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70749cf6-bb66-4d01-b0dc-a06f3ec7ce78-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "70749cf6-bb66-4d01-b0dc-a06f3ec7ce78" (UID: "70749cf6-bb66-4d01-b0dc-a06f3ec7ce78"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:33:22.413812 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:22.413791 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70749cf6-bb66-4d01-b0dc-a06f3ec7ce78-kube-api-access-5j8mp" (OuterVolumeSpecName: "kube-api-access-5j8mp") pod "70749cf6-bb66-4d01-b0dc-a06f3ec7ce78" (UID: "70749cf6-bb66-4d01-b0dc-a06f3ec7ce78"). InnerVolumeSpecName "kube-api-access-5j8mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:33:22.512897 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:22.512868 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70749cf6-bb66-4d01-b0dc-a06f3ec7ce78-extensions-socket-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:33:22.512897 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:22.512893 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j8mp\" (UniqueName: \"kubernetes.io/projected/70749cf6-bb66-4d01-b0dc-a06f3ec7ce78-kube-api-access-5j8mp\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:33:23.085859 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:23.085826 2571 generic.go:356] "Generic (PLEG): container finished" podID="70749cf6-bb66-4d01-b0dc-a06f3ec7ce78" containerID="0919a641ad9c0b643333bc5aaefd1b7f64d11d105f36bdcac3612e5db99d032c" exitCode=0 Jul 06 23:33:23.086265 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:23.085872 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn" event={"ID":"70749cf6-bb66-4d01-b0dc-a06f3ec7ce78","Type":"ContainerDied","Data":"0919a641ad9c0b643333bc5aaefd1b7f64d11d105f36bdcac3612e5db99d032c"} Jul 06 23:33:23.086265 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:23.085910 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn" event={"ID":"70749cf6-bb66-4d01-b0dc-a06f3ec7ce78","Type":"ContainerDied","Data":"a15174a33766e8f7c24231595baa3fbbb12bd884199f2d733897c09aa0e54bf7"} Jul 06 23:33:23.086265 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:23.085930 2571 scope.go:117] "RemoveContainer" containerID="0919a641ad9c0b643333bc5aaefd1b7f64d11d105f36bdcac3612e5db99d032c" Jul 06 23:33:23.086265 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:23.085873 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn" Jul 06 23:33:23.095248 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:23.095231 2571 scope.go:117] "RemoveContainer" containerID="0919a641ad9c0b643333bc5aaefd1b7f64d11d105f36bdcac3612e5db99d032c" Jul 06 23:33:23.095503 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:33:23.095488 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0919a641ad9c0b643333bc5aaefd1b7f64d11d105f36bdcac3612e5db99d032c\": container with ID starting with 0919a641ad9c0b643333bc5aaefd1b7f64d11d105f36bdcac3612e5db99d032c not found: ID does not exist" containerID="0919a641ad9c0b643333bc5aaefd1b7f64d11d105f36bdcac3612e5db99d032c" Jul 06 23:33:23.095545 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:23.095511 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0919a641ad9c0b643333bc5aaefd1b7f64d11d105f36bdcac3612e5db99d032c"} err="failed to get container status \"0919a641ad9c0b643333bc5aaefd1b7f64d11d105f36bdcac3612e5db99d032c\": rpc error: code = NotFound desc = could not find container \"0919a641ad9c0b643333bc5aaefd1b7f64d11d105f36bdcac3612e5db99d032c\": container with ID starting with 0919a641ad9c0b643333bc5aaefd1b7f64d11d105f36bdcac3612e5db99d032c not found: ID does not exist" Jul 06 23:33:23.124071 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:23.124047 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn"] Jul 06 23:33:23.147062 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:23.147035 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-55rfn"] Jul 06 23:33:23.785686 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:23.785647 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70749cf6-bb66-4d01-b0dc-a06f3ec7ce78" path="/var/lib/kubelet/pods/70749cf6-bb66-4d01-b0dc-a06f3ec7ce78/volumes" Jul 06 23:33:26.554160 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.554130 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx"] Jul 06 23:33:26.554619 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.554604 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70749cf6-bb66-4d01-b0dc-a06f3ec7ce78" containerName="manager" Jul 06 23:33:26.554669 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.554621 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="70749cf6-bb66-4d01-b0dc-a06f3ec7ce78" containerName="manager" Jul 06 23:33:26.554705 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.554688 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="70749cf6-bb66-4d01-b0dc-a06f3ec7ce78" containerName="manager" Jul 06 23:33:26.556899 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.556879 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.559419 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.559400 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-x2lj4\"" Jul 06 23:33:26.568928 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.568908 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx"] Jul 06 23:33:26.645483 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.645450 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c72526-90d2-46aa-ac1f-4bd0625ba248-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.645656 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.645504 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c72526-90d2-46aa-ac1f-4bd0625ba248-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.645656 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.645538 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c72526-90d2-46aa-ac1f-4bd0625ba248-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.645656 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.645564 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c72526-90d2-46aa-ac1f-4bd0625ba248-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.645656 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.645614 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vn44\" (UniqueName: \"kubernetes.io/projected/d0c72526-90d2-46aa-ac1f-4bd0625ba248-kube-api-access-8vn44\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.645855 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.645665 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c72526-90d2-46aa-ac1f-4bd0625ba248-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.645855 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.645707 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c72526-90d2-46aa-ac1f-4bd0625ba248-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.645855 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.645779 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c72526-90d2-46aa-ac1f-4bd0625ba248-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.645855 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.645825 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c72526-90d2-46aa-ac1f-4bd0625ba248-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.746698 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.746669 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c72526-90d2-46aa-ac1f-4bd0625ba248-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.746888 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.746711 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c72526-90d2-46aa-ac1f-4bd0625ba248-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.746888 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.746739 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c72526-90d2-46aa-ac1f-4bd0625ba248-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.747005 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.746942 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c72526-90d2-46aa-ac1f-4bd0625ba248-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.747005 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.746992 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c72526-90d2-46aa-ac1f-4bd0625ba248-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.747114 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.747037 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c72526-90d2-46aa-ac1f-4bd0625ba248-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.747114 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.747089 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c72526-90d2-46aa-ac1f-4bd0625ba248-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.747114 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.747093 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c72526-90d2-46aa-ac1f-4bd0625ba248-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.747267 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.747131 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c72526-90d2-46aa-ac1f-4bd0625ba248-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.747267 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.747146 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c72526-90d2-46aa-ac1f-4bd0625ba248-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.747267 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.747159 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vn44\" (UniqueName: \"kubernetes.io/projected/d0c72526-90d2-46aa-ac1f-4bd0625ba248-kube-api-access-8vn44\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.747514 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.747493 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c72526-90d2-46aa-ac1f-4bd0625ba248-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.747648 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.747626 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c72526-90d2-46aa-ac1f-4bd0625ba248-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.747648 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.747639 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c72526-90d2-46aa-ac1f-4bd0625ba248-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.749554 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.749532 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c72526-90d2-46aa-ac1f-4bd0625ba248-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.749671 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.749601 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c72526-90d2-46aa-ac1f-4bd0625ba248-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.755818 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.755763 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c72526-90d2-46aa-ac1f-4bd0625ba248-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.755927 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.755906 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vn44\" (UniqueName: \"kubernetes.io/projected/d0c72526-90d2-46aa-ac1f-4bd0625ba248-kube-api-access-8vn44\") pod \"maas-default-gateway-openshift-default-56654dcf7-24flx\" (UID: \"d0c72526-90d2-46aa-ac1f-4bd0625ba248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:26.869073 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:26.868997 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:27.009125 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:27.009091 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx"] Jul 06 23:33:27.011659 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:33:27.011636 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c72526_90d2_46aa_ac1f_4bd0625ba248.slice/crio-d5e07037101227c0e0529be0d3d502c289a2f0698116c0368533170a8930e658 WatchSource:0}: Error finding container d5e07037101227c0e0529be0d3d502c289a2f0698116c0368533170a8930e658: Status 404 returned error can't find the container with id d5e07037101227c0e0529be0d3d502c289a2f0698116c0368533170a8930e658 Jul 06 23:33:27.013737 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:27.013686 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:33:27.013827 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:27.013804 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:33:27.013889 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:27.013853 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:33:27.107856 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:27.107805 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" event={"ID":"d0c72526-90d2-46aa-ac1f-4bd0625ba248","Type":"ContainerStarted","Data":"59f180746e854fef636832b79ca0670f66fa6770fe7dd97421b9a30762f20650"} Jul 06 23:33:27.108032 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:27.107866 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" event={"ID":"d0c72526-90d2-46aa-ac1f-4bd0625ba248","Type":"ContainerStarted","Data":"d5e07037101227c0e0529be0d3d502c289a2f0698116c0368533170a8930e658"} Jul 06 23:33:27.127873 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:27.127785 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" podStartSLOduration=1.127769824 podStartE2EDuration="1.127769824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:33:27.125892171 +0000 UTC m=+517.839946743" watchObservedRunningTime="2026-07-06 23:33:27.127769824 +0000 UTC m=+517.841824393" Jul 06 23:33:27.869253 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:27.869223 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:28.874919 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:28.874889 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:29.118011 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:29.117974 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:29.119097 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:29.119078 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:33:39.369024 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:39.368986 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-mj4ds"] Jul 06 23:33:39.371215 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:39.371194 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-mj4ds" Jul 06 23:33:39.373815 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:39.373793 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-glwn2\"" Jul 06 23:33:39.378957 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:39.378938 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-mj4ds"] Jul 06 23:33:39.453993 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:39.453961 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzcjn\" (UniqueName: \"kubernetes.io/projected/f870d16e-ce49-4fa8-ad14-3f9b87d771fe-kube-api-access-zzcjn\") pod \"authorino-58cd955675-mj4ds\" (UID: \"f870d16e-ce49-4fa8-ad14-3f9b87d771fe\") " pod="kuadrant-system/authorino-58cd955675-mj4ds" Jul 06 23:33:39.490487 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:39.490458 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-znwnn"] Jul 06 23:33:39.492599 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:39.492584 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-znwnn" Jul 06 23:33:39.504483 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:39.504459 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-znwnn"] Jul 06 23:33:39.554707 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:39.554676 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6nk5\" (UniqueName: \"kubernetes.io/projected/3173fb07-598e-4fd4-8e81-8ce3c2dd72d5-kube-api-access-r6nk5\") pod \"authorino-75d556bd77-znwnn\" (UID: \"3173fb07-598e-4fd4-8e81-8ce3c2dd72d5\") " pod="kuadrant-system/authorino-75d556bd77-znwnn" Jul 06 23:33:39.554855 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:39.554753 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzcjn\" (UniqueName: \"kubernetes.io/projected/f870d16e-ce49-4fa8-ad14-3f9b87d771fe-kube-api-access-zzcjn\") pod \"authorino-58cd955675-mj4ds\" (UID: \"f870d16e-ce49-4fa8-ad14-3f9b87d771fe\") " pod="kuadrant-system/authorino-58cd955675-mj4ds" Jul 06 23:33:39.563018 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:39.562989 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzcjn\" (UniqueName: \"kubernetes.io/projected/f870d16e-ce49-4fa8-ad14-3f9b87d771fe-kube-api-access-zzcjn\") pod \"authorino-58cd955675-mj4ds\" (UID: \"f870d16e-ce49-4fa8-ad14-3f9b87d771fe\") " pod="kuadrant-system/authorino-58cd955675-mj4ds" Jul 06 23:33:39.656089 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:39.656014 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6nk5\" (UniqueName: \"kubernetes.io/projected/3173fb07-598e-4fd4-8e81-8ce3c2dd72d5-kube-api-access-r6nk5\") pod \"authorino-75d556bd77-znwnn\" (UID: \"3173fb07-598e-4fd4-8e81-8ce3c2dd72d5\") " pod="kuadrant-system/authorino-75d556bd77-znwnn" Jul 06 23:33:39.663886 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:39.663854 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6nk5\" (UniqueName: \"kubernetes.io/projected/3173fb07-598e-4fd4-8e81-8ce3c2dd72d5-kube-api-access-r6nk5\") pod \"authorino-75d556bd77-znwnn\" (UID: \"3173fb07-598e-4fd4-8e81-8ce3c2dd72d5\") " pod="kuadrant-system/authorino-75d556bd77-znwnn" Jul 06 23:33:39.681336 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:39.681313 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-mj4ds" Jul 06 23:33:39.800942 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:39.800914 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-znwnn" Jul 06 23:33:39.809934 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:39.809908 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-mj4ds"] Jul 06 23:33:39.812651 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:33:39.812623 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf870d16e_ce49_4fa8_ad14_3f9b87d771fe.slice/crio-cfc4dcd5244af741515cd894a205ffccc4284007ae08fcd30fdd571b6d1fa48e WatchSource:0}: Error finding container cfc4dcd5244af741515cd894a205ffccc4284007ae08fcd30fdd571b6d1fa48e: Status 404 returned error can't find the container with id cfc4dcd5244af741515cd894a205ffccc4284007ae08fcd30fdd571b6d1fa48e Jul 06 23:33:39.936197 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:39.936174 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-znwnn"] Jul 06 23:33:39.938359 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:33:39.938329 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3173fb07_598e_4fd4_8e81_8ce3c2dd72d5.slice/crio-1fe0681fc63fd10dd28f29e7e2ace07d83daba89fa576cdcbda148341142153d WatchSource:0}: Error finding container 1fe0681fc63fd10dd28f29e7e2ace07d83daba89fa576cdcbda148341142153d: Status 404 returned error can't find the container with id 1fe0681fc63fd10dd28f29e7e2ace07d83daba89fa576cdcbda148341142153d Jul 06 23:33:40.173985 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:40.173947 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-mj4ds" event={"ID":"f870d16e-ce49-4fa8-ad14-3f9b87d771fe","Type":"ContainerStarted","Data":"cfc4dcd5244af741515cd894a205ffccc4284007ae08fcd30fdd571b6d1fa48e"} Jul 06 23:33:40.175191 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:40.175168 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-znwnn" event={"ID":"3173fb07-598e-4fd4-8e81-8ce3c2dd72d5","Type":"ContainerStarted","Data":"1fe0681fc63fd10dd28f29e7e2ace07d83daba89fa576cdcbda148341142153d"} Jul 06 23:33:44.204845 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:44.204813 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-mj4ds" event={"ID":"f870d16e-ce49-4fa8-ad14-3f9b87d771fe","Type":"ContainerStarted","Data":"d26fce967eba38d37f920d41cbe41e82e5ca85ebdb45e20c59383fec9fac19e2"} Jul 06 23:33:44.206271 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:44.206247 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-znwnn" event={"ID":"3173fb07-598e-4fd4-8e81-8ce3c2dd72d5","Type":"ContainerStarted","Data":"e05d23082c04f2a0677ed6f366221f1f070322c616814f6ce05ed3257cb885af"} Jul 06 23:33:44.220309 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:44.220258 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-mj4ds" podStartSLOduration=1.863928222 podStartE2EDuration="5.220242487s" totalImagesPullingTime="3.356314265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:33:44.219196912 +0000 UTC m=+534.933251483" watchObservedRunningTime="2026-07-06 23:33:44.220242487 +0000 UTC m=+534.934297057" Jul 06 23:33:44.234405 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:44.234350 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-znwnn" podStartSLOduration=2.015056893 podStartE2EDuration="5.234334454s" totalImagesPullingTime="3.219277561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:33:44.232910312 +0000 UTC m=+534.946964882" watchObservedRunningTime="2026-07-06 23:33:44.234334454 +0000 UTC m=+534.948389023" Jul 06 23:33:44.260289 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:44.260256 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-mj4ds"] Jul 06 23:33:46.216185 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:46.216127 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-mj4ds" podUID="f870d16e-ce49-4fa8-ad14-3f9b87d771fe" containerName="authorino" containerID="cri-o://d26fce967eba38d37f920d41cbe41e82e5ca85ebdb45e20c59383fec9fac19e2" gracePeriod=30 Jul 06 23:33:46.467215 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:46.467162 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-mj4ds" Jul 06 23:33:46.620183 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:46.620152 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzcjn\" (UniqueName: \"kubernetes.io/projected/f870d16e-ce49-4fa8-ad14-3f9b87d771fe-kube-api-access-zzcjn\") pod \"f870d16e-ce49-4fa8-ad14-3f9b87d771fe\" (UID: \"f870d16e-ce49-4fa8-ad14-3f9b87d771fe\") " Jul 06 23:33:46.622417 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:46.622391 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f870d16e-ce49-4fa8-ad14-3f9b87d771fe-kube-api-access-zzcjn" (OuterVolumeSpecName: "kube-api-access-zzcjn") pod "f870d16e-ce49-4fa8-ad14-3f9b87d771fe" (UID: "f870d16e-ce49-4fa8-ad14-3f9b87d771fe"). InnerVolumeSpecName "kube-api-access-zzcjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:33:46.720948 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:46.720886 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzcjn\" (UniqueName: \"kubernetes.io/projected/f870d16e-ce49-4fa8-ad14-3f9b87d771fe-kube-api-access-zzcjn\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:33:47.222401 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:47.222368 2571 generic.go:356] "Generic (PLEG): container finished" podID="f870d16e-ce49-4fa8-ad14-3f9b87d771fe" containerID="d26fce967eba38d37f920d41cbe41e82e5ca85ebdb45e20c59383fec9fac19e2" exitCode=0 Jul 06 23:33:47.222852 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:47.222418 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-mj4ds" Jul 06 23:33:47.222852 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:47.222448 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-mj4ds" event={"ID":"f870d16e-ce49-4fa8-ad14-3f9b87d771fe","Type":"ContainerDied","Data":"d26fce967eba38d37f920d41cbe41e82e5ca85ebdb45e20c59383fec9fac19e2"} Jul 06 23:33:47.222852 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:47.222487 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-mj4ds" event={"ID":"f870d16e-ce49-4fa8-ad14-3f9b87d771fe","Type":"ContainerDied","Data":"cfc4dcd5244af741515cd894a205ffccc4284007ae08fcd30fdd571b6d1fa48e"} Jul 06 23:33:47.222852 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:47.222503 2571 scope.go:117] "RemoveContainer" containerID="d26fce967eba38d37f920d41cbe41e82e5ca85ebdb45e20c59383fec9fac19e2" Jul 06 23:33:47.231496 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:47.231481 2571 scope.go:117] "RemoveContainer" containerID="d26fce967eba38d37f920d41cbe41e82e5ca85ebdb45e20c59383fec9fac19e2" Jul 06 23:33:47.231782 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:33:47.231707 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26fce967eba38d37f920d41cbe41e82e5ca85ebdb45e20c59383fec9fac19e2\": container with ID starting with d26fce967eba38d37f920d41cbe41e82e5ca85ebdb45e20c59383fec9fac19e2 not found: ID does not exist" containerID="d26fce967eba38d37f920d41cbe41e82e5ca85ebdb45e20c59383fec9fac19e2" Jul 06 23:33:47.231840 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:47.231790 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26fce967eba38d37f920d41cbe41e82e5ca85ebdb45e20c59383fec9fac19e2"} err="failed to get container status \"d26fce967eba38d37f920d41cbe41e82e5ca85ebdb45e20c59383fec9fac19e2\": rpc error: code = NotFound desc = could not find container \"d26fce967eba38d37f920d41cbe41e82e5ca85ebdb45e20c59383fec9fac19e2\": container with ID starting with d26fce967eba38d37f920d41cbe41e82e5ca85ebdb45e20c59383fec9fac19e2 not found: ID does not exist" Jul 06 23:33:47.243322 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:47.243294 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-mj4ds"] Jul 06 23:33:47.249020 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:47.249001 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-mj4ds"] Jul 06 23:33:47.785929 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:33:47.785900 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f870d16e-ce49-4fa8-ad14-3f9b87d771fe" path="/var/lib/kubelet/pods/f870d16e-ce49-4fa8-ad14-3f9b87d771fe/volumes" Jul 06 23:34:07.801257 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:07.801224 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zc9xn"] Jul 06 23:34:07.801706 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:07.801569 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f870d16e-ce49-4fa8-ad14-3f9b87d771fe" containerName="authorino" Jul 06 23:34:07.801706 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:07.801586 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f870d16e-ce49-4fa8-ad14-3f9b87d771fe" containerName="authorino" Jul 06 23:34:07.801706 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:07.801655 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f870d16e-ce49-4fa8-ad14-3f9b87d771fe" containerName="authorino" Jul 06 23:34:07.807034 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:07.807013 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zc9xn" Jul 06 23:34:07.815536 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:07.815515 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zc9xn"] Jul 06 23:34:07.885573 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:07.885543 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwmsm\" (UniqueName: \"kubernetes.io/projected/80ebf1f0-3745-45b6-a2ba-2f9b06941f54-kube-api-access-pwmsm\") pod \"authorino-5d4b67c446-zc9xn\" (UID: \"80ebf1f0-3745-45b6-a2ba-2f9b06941f54\") " pod="kuadrant-system/authorino-5d4b67c446-zc9xn" Jul 06 23:34:07.986175 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:07.986144 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwmsm\" (UniqueName: \"kubernetes.io/projected/80ebf1f0-3745-45b6-a2ba-2f9b06941f54-kube-api-access-pwmsm\") pod \"authorino-5d4b67c446-zc9xn\" (UID: \"80ebf1f0-3745-45b6-a2ba-2f9b06941f54\") " pod="kuadrant-system/authorino-5d4b67c446-zc9xn" Jul 06 23:34:07.993402 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:07.993378 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zc9xn"] Jul 06 23:34:07.993585 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:34:07.993568 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-pwmsm], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-zc9xn" podUID="80ebf1f0-3745-45b6-a2ba-2f9b06941f54" Jul 06 23:34:07.996989 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:07.996972 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwmsm\" (UniqueName: \"kubernetes.io/projected/80ebf1f0-3745-45b6-a2ba-2f9b06941f54-kube-api-access-pwmsm\") pod \"authorino-5d4b67c446-zc9xn\" (UID: \"80ebf1f0-3745-45b6-a2ba-2f9b06941f54\") " pod="kuadrant-system/authorino-5d4b67c446-zc9xn" Jul 06 23:34:08.060451 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.060386 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-5blx8"] Jul 06 23:34:08.062858 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.062842 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-5blx8" Jul 06 23:34:08.068802 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.068783 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 23:34:08.079162 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.079141 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-5blx8"] Jul 06 23:34:08.087383 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.087351 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44-tls-cert\") pod \"authorino-76877994-5blx8\" (UID: \"14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44\") " pod="kuadrant-system/authorino-76877994-5blx8" Jul 06 23:34:08.087477 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.087393 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwnjw\" (UniqueName: \"kubernetes.io/projected/14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44-kube-api-access-mwnjw\") pod \"authorino-76877994-5blx8\" (UID: \"14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44\") " pod="kuadrant-system/authorino-76877994-5blx8" Jul 06 23:34:08.187926 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.187895 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44-tls-cert\") pod \"authorino-76877994-5blx8\" (UID: \"14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44\") " pod="kuadrant-system/authorino-76877994-5blx8" Jul 06 23:34:08.188065 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.187934 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwnjw\" (UniqueName: \"kubernetes.io/projected/14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44-kube-api-access-mwnjw\") pod \"authorino-76877994-5blx8\" (UID: \"14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44\") " pod="kuadrant-system/authorino-76877994-5blx8" Jul 06 23:34:08.190534 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.190512 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44-tls-cert\") pod \"authorino-76877994-5blx8\" (UID: \"14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44\") " pod="kuadrant-system/authorino-76877994-5blx8" Jul 06 23:34:08.200940 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.200916 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwnjw\" (UniqueName: \"kubernetes.io/projected/14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44-kube-api-access-mwnjw\") pod \"authorino-76877994-5blx8\" (UID: \"14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44\") " pod="kuadrant-system/authorino-76877994-5blx8" Jul 06 23:34:08.231314 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.231289 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-5blx8"] Jul 06 23:34:08.231497 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.231485 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-5blx8" Jul 06 23:34:08.262208 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.262181 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f6cbb5bdd-k8drl"] Jul 06 23:34:08.264643 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.264624 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f6cbb5bdd-k8drl" Jul 06 23:34:08.277864 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.277843 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f6cbb5bdd-k8drl"] Jul 06 23:34:08.288865 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.288837 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnpjl\" (UniqueName: \"kubernetes.io/projected/7e68140a-2fb6-406d-a185-0ba44acccf50-kube-api-access-pnpjl\") pod \"authorino-7f6cbb5bdd-k8drl\" (UID: \"7e68140a-2fb6-406d-a185-0ba44acccf50\") " pod="kuadrant-system/authorino-7f6cbb5bdd-k8drl" Jul 06 23:34:08.288978 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.288908 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e68140a-2fb6-406d-a185-0ba44acccf50-tls-cert\") pod \"authorino-7f6cbb5bdd-k8drl\" (UID: \"7e68140a-2fb6-406d-a185-0ba44acccf50\") " pod="kuadrant-system/authorino-7f6cbb5bdd-k8drl" Jul 06 23:34:08.328211 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.328184 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zc9xn" Jul 06 23:34:08.334092 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.334067 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zc9xn" Jul 06 23:34:08.367811 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.367784 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-5blx8"] Jul 06 23:34:08.370634 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:34:08.370612 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14ef6b7f_a80f_42f6_afdf_3f4b1bdc4e44.slice/crio-aee9ab1ef24ddf3e3ab5c099161fe3f212d7d80c55623f26a561138593ac6df6 WatchSource:0}: Error finding container aee9ab1ef24ddf3e3ab5c099161fe3f212d7d80c55623f26a561138593ac6df6: Status 404 returned error can't find the container with id aee9ab1ef24ddf3e3ab5c099161fe3f212d7d80c55623f26a561138593ac6df6 Jul 06 23:34:08.389789 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.389763 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwmsm\" (UniqueName: \"kubernetes.io/projected/80ebf1f0-3745-45b6-a2ba-2f9b06941f54-kube-api-access-pwmsm\") pod \"80ebf1f0-3745-45b6-a2ba-2f9b06941f54\" (UID: \"80ebf1f0-3745-45b6-a2ba-2f9b06941f54\") " Jul 06 23:34:08.389913 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.389897 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e68140a-2fb6-406d-a185-0ba44acccf50-tls-cert\") pod \"authorino-7f6cbb5bdd-k8drl\" (UID: \"7e68140a-2fb6-406d-a185-0ba44acccf50\") " pod="kuadrant-system/authorino-7f6cbb5bdd-k8drl" Jul 06 23:34:08.390003 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.389988 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnpjl\" (UniqueName: \"kubernetes.io/projected/7e68140a-2fb6-406d-a185-0ba44acccf50-kube-api-access-pnpjl\") pod \"authorino-7f6cbb5bdd-k8drl\" (UID: \"7e68140a-2fb6-406d-a185-0ba44acccf50\") " pod="kuadrant-system/authorino-7f6cbb5bdd-k8drl" Jul 06 23:34:08.391892 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.391871 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ebf1f0-3745-45b6-a2ba-2f9b06941f54-kube-api-access-pwmsm" (OuterVolumeSpecName: "kube-api-access-pwmsm") pod "80ebf1f0-3745-45b6-a2ba-2f9b06941f54" (UID: "80ebf1f0-3745-45b6-a2ba-2f9b06941f54"). InnerVolumeSpecName "kube-api-access-pwmsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:34:08.392262 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.392246 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e68140a-2fb6-406d-a185-0ba44acccf50-tls-cert\") pod \"authorino-7f6cbb5bdd-k8drl\" (UID: \"7e68140a-2fb6-406d-a185-0ba44acccf50\") " pod="kuadrant-system/authorino-7f6cbb5bdd-k8drl" Jul 06 23:34:08.398560 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.398537 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnpjl\" (UniqueName: \"kubernetes.io/projected/7e68140a-2fb6-406d-a185-0ba44acccf50-kube-api-access-pnpjl\") pod \"authorino-7f6cbb5bdd-k8drl\" (UID: \"7e68140a-2fb6-406d-a185-0ba44acccf50\") " pod="kuadrant-system/authorino-7f6cbb5bdd-k8drl" Jul 06 23:34:08.490332 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.490306 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwmsm\" (UniqueName: \"kubernetes.io/projected/80ebf1f0-3745-45b6-a2ba-2f9b06941f54-kube-api-access-pwmsm\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:34:08.574623 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.574569 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f6cbb5bdd-k8drl" Jul 06 23:34:08.710193 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:08.710165 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f6cbb5bdd-k8drl"] Jul 06 23:34:08.710932 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:34:08.710913 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e68140a_2fb6_406d_a185_0ba44acccf50.slice/crio-8f5bde8b61557a1b2714a71da7f1eb1b31bf6bedc7fca98ca95659891f90b249 WatchSource:0}: Error finding container 8f5bde8b61557a1b2714a71da7f1eb1b31bf6bedc7fca98ca95659891f90b249: Status 404 returned error can't find the container with id 8f5bde8b61557a1b2714a71da7f1eb1b31bf6bedc7fca98ca95659891f90b249 Jul 06 23:34:09.335017 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.334956 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76877994-5blx8" event={"ID":"14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44","Type":"ContainerStarted","Data":"96245a08de3bf3b3ede9ba7b8d262b1147f5f7219c08fc2a8724bce6ee35d299"} Jul 06 23:34:09.335017 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.335004 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76877994-5blx8" event={"ID":"14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44","Type":"ContainerStarted","Data":"aee9ab1ef24ddf3e3ab5c099161fe3f212d7d80c55623f26a561138593ac6df6"} Jul 06 23:34:09.335017 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.335001 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-76877994-5blx8" podUID="14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44" containerName="authorino" containerID="cri-o://96245a08de3bf3b3ede9ba7b8d262b1147f5f7219c08fc2a8724bce6ee35d299" gracePeriod=30 Jul 06 23:34:09.337059 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.337035 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zc9xn" Jul 06 23:34:09.337172 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.337036 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f6cbb5bdd-k8drl" event={"ID":"7e68140a-2fb6-406d-a185-0ba44acccf50","Type":"ContainerStarted","Data":"55b907224719956f7e5b1478f4f6a62b835f6dae31d02c92fda191f6d80e7a3a"} Jul 06 23:34:09.337172 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.337151 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f6cbb5bdd-k8drl" event={"ID":"7e68140a-2fb6-406d-a185-0ba44acccf50","Type":"ContainerStarted","Data":"8f5bde8b61557a1b2714a71da7f1eb1b31bf6bedc7fca98ca95659891f90b249"} Jul 06 23:34:09.352840 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.352799 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-76877994-5blx8" podStartSLOduration=1.075181899 podStartE2EDuration="1.352784292s" totalImagesPullingTime="277.602393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:34:09.35140361 +0000 UTC m=+560.065458181" watchObservedRunningTime="2026-07-06 23:34:09.352784292 +0000 UTC m=+560.066838863" Jul 06 23:34:09.368270 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.368223 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7f6cbb5bdd-k8drl" podStartSLOduration=0.844855413 podStartE2EDuration="1.368207949s" totalImagesPullingTime="523.352536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:34:09.366200366 +0000 UTC m=+560.080254936" watchObservedRunningTime="2026-07-06 23:34:09.368207949 +0000 UTC m=+560.082262510" Jul 06 23:34:09.394914 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.394879 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-znwnn"] Jul 06 23:34:09.395139 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.395104 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-znwnn" podUID="3173fb07-598e-4fd4-8e81-8ce3c2dd72d5" containerName="authorino" containerID="cri-o://e05d23082c04f2a0677ed6f366221f1f070322c616814f6ce05ed3257cb885af" gracePeriod=30 Jul 06 23:34:09.474326 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.474283 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zc9xn"] Jul 06 23:34:09.476768 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.476744 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zc9xn"] Jul 06 23:34:09.638512 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.638485 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-5blx8" Jul 06 23:34:09.666239 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.666214 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-znwnn" Jul 06 23:34:09.699326 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.699297 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44-tls-cert\") pod \"14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44\" (UID: \"14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44\") " Jul 06 23:34:09.699326 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.699326 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwnjw\" (UniqueName: \"kubernetes.io/projected/14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44-kube-api-access-mwnjw\") pod \"14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44\" (UID: \"14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44\") " Jul 06 23:34:09.699535 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.699365 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6nk5\" (UniqueName: \"kubernetes.io/projected/3173fb07-598e-4fd4-8e81-8ce3c2dd72d5-kube-api-access-r6nk5\") pod \"3173fb07-598e-4fd4-8e81-8ce3c2dd72d5\" (UID: \"3173fb07-598e-4fd4-8e81-8ce3c2dd72d5\") " Jul 06 23:34:09.701432 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.701399 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3173fb07-598e-4fd4-8e81-8ce3c2dd72d5-kube-api-access-r6nk5" (OuterVolumeSpecName: "kube-api-access-r6nk5") pod "3173fb07-598e-4fd4-8e81-8ce3c2dd72d5" (UID: "3173fb07-598e-4fd4-8e81-8ce3c2dd72d5"). InnerVolumeSpecName "kube-api-access-r6nk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:34:09.701526 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.701455 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44-kube-api-access-mwnjw" (OuterVolumeSpecName: "kube-api-access-mwnjw") pod "14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44" (UID: "14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44"). InnerVolumeSpecName "kube-api-access-mwnjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:34:09.709635 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.709608 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44" (UID: "14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:34:09.786301 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.786268 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80ebf1f0-3745-45b6-a2ba-2f9b06941f54" path="/var/lib/kubelet/pods/80ebf1f0-3745-45b6-a2ba-2f9b06941f54/volumes" Jul 06 23:34:09.800005 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.799983 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6nk5\" (UniqueName: \"kubernetes.io/projected/3173fb07-598e-4fd4-8e81-8ce3c2dd72d5-kube-api-access-r6nk5\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:34:09.800090 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.800007 2571 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44-tls-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:34:09.800090 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:09.800019 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwnjw\" (UniqueName: \"kubernetes.io/projected/14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44-kube-api-access-mwnjw\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:34:10.343096 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:10.343009 2571 generic.go:356] "Generic (PLEG): container finished" podID="3173fb07-598e-4fd4-8e81-8ce3c2dd72d5" containerID="e05d23082c04f2a0677ed6f366221f1f070322c616814f6ce05ed3257cb885af" exitCode=0 Jul 06 23:34:10.343096 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:10.343067 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-znwnn" Jul 06 23:34:10.343571 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:10.343102 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-znwnn" event={"ID":"3173fb07-598e-4fd4-8e81-8ce3c2dd72d5","Type":"ContainerDied","Data":"e05d23082c04f2a0677ed6f366221f1f070322c616814f6ce05ed3257cb885af"} Jul 06 23:34:10.343571 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:10.343151 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-znwnn" event={"ID":"3173fb07-598e-4fd4-8e81-8ce3c2dd72d5","Type":"ContainerDied","Data":"1fe0681fc63fd10dd28f29e7e2ace07d83daba89fa576cdcbda148341142153d"} Jul 06 23:34:10.343571 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:10.343171 2571 scope.go:117] "RemoveContainer" containerID="e05d23082c04f2a0677ed6f366221f1f070322c616814f6ce05ed3257cb885af" Jul 06 23:34:10.344515 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:10.344450 2571 generic.go:356] "Generic (PLEG): container finished" podID="14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44" containerID="96245a08de3bf3b3ede9ba7b8d262b1147f5f7219c08fc2a8724bce6ee35d299" exitCode=0 Jul 06 23:34:10.344515 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:10.344500 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-5blx8" Jul 06 23:34:10.344645 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:10.344543 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76877994-5blx8" event={"ID":"14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44","Type":"ContainerDied","Data":"96245a08de3bf3b3ede9ba7b8d262b1147f5f7219c08fc2a8724bce6ee35d299"} Jul 06 23:34:10.344645 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:10.344581 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76877994-5blx8" event={"ID":"14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44","Type":"ContainerDied","Data":"aee9ab1ef24ddf3e3ab5c099161fe3f212d7d80c55623f26a561138593ac6df6"} Jul 06 23:34:10.352243 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:10.352226 2571 scope.go:117] "RemoveContainer" containerID="e05d23082c04f2a0677ed6f366221f1f070322c616814f6ce05ed3257cb885af" Jul 06 23:34:10.352503 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:34:10.352487 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05d23082c04f2a0677ed6f366221f1f070322c616814f6ce05ed3257cb885af\": container with ID starting with e05d23082c04f2a0677ed6f366221f1f070322c616814f6ce05ed3257cb885af not found: ID does not exist" containerID="e05d23082c04f2a0677ed6f366221f1f070322c616814f6ce05ed3257cb885af" Jul 06 23:34:10.352557 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:10.352510 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05d23082c04f2a0677ed6f366221f1f070322c616814f6ce05ed3257cb885af"} err="failed to get container status \"e05d23082c04f2a0677ed6f366221f1f070322c616814f6ce05ed3257cb885af\": rpc error: code = NotFound desc = could not find container \"e05d23082c04f2a0677ed6f366221f1f070322c616814f6ce05ed3257cb885af\": container with ID starting with e05d23082c04f2a0677ed6f366221f1f070322c616814f6ce05ed3257cb885af not found: ID does not exist" Jul 06 23:34:10.352557 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:10.352524 2571 scope.go:117] "RemoveContainer" containerID="96245a08de3bf3b3ede9ba7b8d262b1147f5f7219c08fc2a8724bce6ee35d299" Jul 06 23:34:10.360284 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:10.360258 2571 scope.go:117] "RemoveContainer" containerID="96245a08de3bf3b3ede9ba7b8d262b1147f5f7219c08fc2a8724bce6ee35d299" Jul 06 23:34:10.360794 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:34:10.360768 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96245a08de3bf3b3ede9ba7b8d262b1147f5f7219c08fc2a8724bce6ee35d299\": container with ID starting with 96245a08de3bf3b3ede9ba7b8d262b1147f5f7219c08fc2a8724bce6ee35d299 not found: ID does not exist" containerID="96245a08de3bf3b3ede9ba7b8d262b1147f5f7219c08fc2a8724bce6ee35d299" Jul 06 23:34:10.360909 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:10.360802 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96245a08de3bf3b3ede9ba7b8d262b1147f5f7219c08fc2a8724bce6ee35d299"} err="failed to get container status \"96245a08de3bf3b3ede9ba7b8d262b1147f5f7219c08fc2a8724bce6ee35d299\": rpc error: code = NotFound desc = could not find container \"96245a08de3bf3b3ede9ba7b8d262b1147f5f7219c08fc2a8724bce6ee35d299\": container with ID starting with 96245a08de3bf3b3ede9ba7b8d262b1147f5f7219c08fc2a8724bce6ee35d299 not found: ID does not exist" Jul 06 23:34:10.362683 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:10.362665 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-znwnn"] Jul 06 23:34:10.370016 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:10.369992 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-znwnn"] Jul 06 23:34:10.382861 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:10.382836 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-5blx8"] Jul 06 23:34:10.386462 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:10.386442 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-5blx8"] Jul 06 23:34:11.786035 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:11.785992 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44" path="/var/lib/kubelet/pods/14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44/volumes" Jul 06 23:34:11.786517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:11.786450 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3173fb07-598e-4fd4-8e81-8ce3c2dd72d5" path="/var/lib/kubelet/pods/3173fb07-598e-4fd4-8e81-8ce3c2dd72d5/volumes" Jul 06 23:34:24.339006 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.338971 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-869c4665f8-cdpzt"] Jul 06 23:34:24.339455 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.339428 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3173fb07-598e-4fd4-8e81-8ce3c2dd72d5" containerName="authorino" Jul 06 23:34:24.339455 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.339442 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3173fb07-598e-4fd4-8e81-8ce3c2dd72d5" containerName="authorino" Jul 06 23:34:24.339530 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.339462 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44" containerName="authorino" Jul 06 23:34:24.339530 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.339470 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44" containerName="authorino" Jul 06 23:34:24.339590 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.339541 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="14ef6b7f-a80f-42f6-afdf-3f4b1bdc4e44" containerName="authorino" Jul 06 23:34:24.339590 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.339555 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3173fb07-598e-4fd4-8e81-8ce3c2dd72d5" containerName="authorino" Jul 06 23:34:24.341861 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.341841 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-869c4665f8-cdpzt" Jul 06 23:34:24.344298 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.344273 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 23:34:24.344473 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.344388 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 23:34:24.345164 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.345147 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mfnr2\"" Jul 06 23:34:24.353799 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.353778 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-869c4665f8-cdpzt"] Jul 06 23:34:24.415019 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.414988 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/520bfe39-f561-411d-9c5b-301cf5d73de7-webhook-cert\") pod \"maas-controller-869c4665f8-cdpzt\" (UID: \"520bfe39-f561-411d-9c5b-301cf5d73de7\") " pod="opendatahub/maas-controller-869c4665f8-cdpzt" Jul 06 23:34:24.415200 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.415049 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44dkv\" (UniqueName: \"kubernetes.io/projected/520bfe39-f561-411d-9c5b-301cf5d73de7-kube-api-access-44dkv\") pod \"maas-controller-869c4665f8-cdpzt\" (UID: \"520bfe39-f561-411d-9c5b-301cf5d73de7\") " pod="opendatahub/maas-controller-869c4665f8-cdpzt" Jul 06 23:34:24.515979 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.515944 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44dkv\" (UniqueName: \"kubernetes.io/projected/520bfe39-f561-411d-9c5b-301cf5d73de7-kube-api-access-44dkv\") pod \"maas-controller-869c4665f8-cdpzt\" (UID: \"520bfe39-f561-411d-9c5b-301cf5d73de7\") " pod="opendatahub/maas-controller-869c4665f8-cdpzt" Jul 06 23:34:24.516148 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.516006 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/520bfe39-f561-411d-9c5b-301cf5d73de7-webhook-cert\") pod \"maas-controller-869c4665f8-cdpzt\" (UID: \"520bfe39-f561-411d-9c5b-301cf5d73de7\") " pod="opendatahub/maas-controller-869c4665f8-cdpzt" Jul 06 23:34:24.518649 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.518613 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/520bfe39-f561-411d-9c5b-301cf5d73de7-webhook-cert\") pod \"maas-controller-869c4665f8-cdpzt\" (UID: \"520bfe39-f561-411d-9c5b-301cf5d73de7\") " pod="opendatahub/maas-controller-869c4665f8-cdpzt" Jul 06 23:34:24.529422 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.529393 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44dkv\" (UniqueName: \"kubernetes.io/projected/520bfe39-f561-411d-9c5b-301cf5d73de7-kube-api-access-44dkv\") pod \"maas-controller-869c4665f8-cdpzt\" (UID: \"520bfe39-f561-411d-9c5b-301cf5d73de7\") " pod="opendatahub/maas-controller-869c4665f8-cdpzt" Jul 06 23:34:24.653098 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.653011 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-869c4665f8-cdpzt" Jul 06 23:34:24.786165 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:24.786136 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-869c4665f8-cdpzt"] Jul 06 23:34:24.788289 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:34:24.788261 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520bfe39_f561_411d_9c5b_301cf5d73de7.slice/crio-1cabd93ca49a89d08b787333824f08cb37ae29a7f9afacfb3abb928f65a0e85f WatchSource:0}: Error finding container 1cabd93ca49a89d08b787333824f08cb37ae29a7f9afacfb3abb928f65a0e85f: Status 404 returned error can't find the container with id 1cabd93ca49a89d08b787333824f08cb37ae29a7f9afacfb3abb928f65a0e85f Jul 06 23:34:25.424536 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:25.424459 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-869c4665f8-cdpzt" event={"ID":"520bfe39-f561-411d-9c5b-301cf5d73de7","Type":"ContainerStarted","Data":"1cabd93ca49a89d08b787333824f08cb37ae29a7f9afacfb3abb928f65a0e85f"} Jul 06 23:34:27.436501 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:27.436464 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-869c4665f8-cdpzt" event={"ID":"520bfe39-f561-411d-9c5b-301cf5d73de7","Type":"ContainerStarted","Data":"9072bb5fd616ee7e400797068802b8871cc29b2ffd74e339c27374bb35b96f13"} Jul 06 23:34:27.436924 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:27.436602 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-869c4665f8-cdpzt" Jul 06 23:34:27.453666 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:27.453593 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-869c4665f8-cdpzt" podStartSLOduration=1.676515183 podStartE2EDuration="3.453578265s" totalImagesPullingTime="1.777063082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:34:27.452478389 +0000 UTC m=+578.166532960" watchObservedRunningTime="2026-07-06 23:34:27.453578265 +0000 UTC m=+578.167632834" Jul 06 23:34:38.447142 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:38.447106 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-869c4665f8-cdpzt" Jul 06 23:34:39.356676 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:39.356640 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-869c4665f8-cdpzt"] Jul 06 23:34:39.356861 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:39.356842 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-869c4665f8-cdpzt" podUID="520bfe39-f561-411d-9c5b-301cf5d73de7" containerName="manager" containerID="cri-o://9072bb5fd616ee7e400797068802b8871cc29b2ffd74e339c27374bb35b96f13" gracePeriod=30 Jul 06 23:34:39.502022 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:39.501987 2571 generic.go:356] "Generic (PLEG): container finished" podID="520bfe39-f561-411d-9c5b-301cf5d73de7" containerID="9072bb5fd616ee7e400797068802b8871cc29b2ffd74e339c27374bb35b96f13" exitCode=0 Jul 06 23:34:39.502385 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:39.502043 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-869c4665f8-cdpzt" event={"ID":"520bfe39-f561-411d-9c5b-301cf5d73de7","Type":"ContainerDied","Data":"9072bb5fd616ee7e400797068802b8871cc29b2ffd74e339c27374bb35b96f13"} Jul 06 23:34:39.593641 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:39.593618 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-869c4665f8-cdpzt" Jul 06 23:34:39.641802 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:39.641732 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/520bfe39-f561-411d-9c5b-301cf5d73de7-webhook-cert\") pod \"520bfe39-f561-411d-9c5b-301cf5d73de7\" (UID: \"520bfe39-f561-411d-9c5b-301cf5d73de7\") " Jul 06 23:34:39.641933 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:39.641807 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44dkv\" (UniqueName: \"kubernetes.io/projected/520bfe39-f561-411d-9c5b-301cf5d73de7-kube-api-access-44dkv\") pod \"520bfe39-f561-411d-9c5b-301cf5d73de7\" (UID: \"520bfe39-f561-411d-9c5b-301cf5d73de7\") " Jul 06 23:34:39.643772 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:39.643748 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/520bfe39-f561-411d-9c5b-301cf5d73de7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "520bfe39-f561-411d-9c5b-301cf5d73de7" (UID: "520bfe39-f561-411d-9c5b-301cf5d73de7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:34:39.643910 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:39.643890 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520bfe39-f561-411d-9c5b-301cf5d73de7-kube-api-access-44dkv" (OuterVolumeSpecName: "kube-api-access-44dkv") pod "520bfe39-f561-411d-9c5b-301cf5d73de7" (UID: "520bfe39-f561-411d-9c5b-301cf5d73de7"). InnerVolumeSpecName "kube-api-access-44dkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:34:39.742553 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:39.742523 2571 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/520bfe39-f561-411d-9c5b-301cf5d73de7-webhook-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:34:39.742553 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:39.742551 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44dkv\" (UniqueName: \"kubernetes.io/projected/520bfe39-f561-411d-9c5b-301cf5d73de7-kube-api-access-44dkv\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:34:40.508431 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:40.508392 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-869c4665f8-cdpzt" event={"ID":"520bfe39-f561-411d-9c5b-301cf5d73de7","Type":"ContainerDied","Data":"1cabd93ca49a89d08b787333824f08cb37ae29a7f9afacfb3abb928f65a0e85f"} Jul 06 23:34:40.508431 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:40.508429 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-869c4665f8-cdpzt" Jul 06 23:34:40.508968 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:40.508473 2571 scope.go:117] "RemoveContainer" containerID="9072bb5fd616ee7e400797068802b8871cc29b2ffd74e339c27374bb35b96f13" Jul 06 23:34:40.524075 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:40.524051 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-869c4665f8-cdpzt"] Jul 06 23:34:40.527367 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:40.527341 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-869c4665f8-cdpzt"] Jul 06 23:34:41.784863 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:41.784834 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="520bfe39-f561-411d-9c5b-301cf5d73de7" path="/var/lib/kubelet/pods/520bfe39-f561-411d-9c5b-301cf5d73de7/volumes" Jul 06 23:34:49.687817 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:49.687782 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 06 23:34:49.689456 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:34:49.689425 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 06 23:35:00.026516 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:00.026484 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-6ph7p"] Jul 06 23:35:00.029834 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:00.026811 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="520bfe39-f561-411d-9c5b-301cf5d73de7" containerName="manager" Jul 06 23:35:00.029834 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:00.026822 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="520bfe39-f561-411d-9c5b-301cf5d73de7" containerName="manager" Jul 06 23:35:00.029834 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:00.026885 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="520bfe39-f561-411d-9c5b-301cf5d73de7" containerName="manager" Jul 06 23:35:00.031074 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:00.031056 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-6ph7p" Jul 06 23:35:00.033362 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:00.033342 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-q8tq2\"" Jul 06 23:35:00.033908 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:00.033893 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 23:35:00.041019 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:00.040999 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-6ph7p"] Jul 06 23:35:00.204293 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:00.204256 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d948f999-2d98-4c3e-92fa-19cef82771ae-plugins-config-volume\") pod \"payload-pre-processing-7964457696-6ph7p\" (UID: \"d948f999-2d98-4c3e-92fa-19cef82771ae\") " pod="openshift-ingress/payload-pre-processing-7964457696-6ph7p" Jul 06 23:35:00.204468 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:00.204355 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxsc6\" (UniqueName: \"kubernetes.io/projected/d948f999-2d98-4c3e-92fa-19cef82771ae-kube-api-access-fxsc6\") pod \"payload-pre-processing-7964457696-6ph7p\" (UID: \"d948f999-2d98-4c3e-92fa-19cef82771ae\") " pod="openshift-ingress/payload-pre-processing-7964457696-6ph7p" Jul 06 23:35:00.305742 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:00.305625 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxsc6\" (UniqueName: \"kubernetes.io/projected/d948f999-2d98-4c3e-92fa-19cef82771ae-kube-api-access-fxsc6\") pod \"payload-pre-processing-7964457696-6ph7p\" (UID: \"d948f999-2d98-4c3e-92fa-19cef82771ae\") " pod="openshift-ingress/payload-pre-processing-7964457696-6ph7p" Jul 06 23:35:00.305906 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:00.305745 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d948f999-2d98-4c3e-92fa-19cef82771ae-plugins-config-volume\") pod \"payload-pre-processing-7964457696-6ph7p\" (UID: \"d948f999-2d98-4c3e-92fa-19cef82771ae\") " pod="openshift-ingress/payload-pre-processing-7964457696-6ph7p" Jul 06 23:35:00.306437 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:00.306413 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d948f999-2d98-4c3e-92fa-19cef82771ae-plugins-config-volume\") pod \"payload-pre-processing-7964457696-6ph7p\" (UID: \"d948f999-2d98-4c3e-92fa-19cef82771ae\") " pod="openshift-ingress/payload-pre-processing-7964457696-6ph7p" Jul 06 23:35:00.316258 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:00.316218 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxsc6\" (UniqueName: \"kubernetes.io/projected/d948f999-2d98-4c3e-92fa-19cef82771ae-kube-api-access-fxsc6\") pod \"payload-pre-processing-7964457696-6ph7p\" (UID: \"d948f999-2d98-4c3e-92fa-19cef82771ae\") " pod="openshift-ingress/payload-pre-processing-7964457696-6ph7p" Jul 06 23:35:00.342103 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:00.342076 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-6ph7p" Jul 06 23:35:00.474509 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:00.474487 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-6ph7p"] Jul 06 23:35:00.477009 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:35:00.476984 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd948f999_2d98_4c3e_92fa_19cef82771ae.slice/crio-89d175dd8675eb83a0af137a5789025fc89fd6eecced4aeec664f768661099bc WatchSource:0}: Error finding container 89d175dd8675eb83a0af137a5789025fc89fd6eecced4aeec664f768661099bc: Status 404 returned error can't find the container with id 89d175dd8675eb83a0af137a5789025fc89fd6eecced4aeec664f768661099bc Jul 06 23:35:00.607777 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:00.607681 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-6ph7p" event={"ID":"d948f999-2d98-4c3e-92fa-19cef82771ae","Type":"ContainerStarted","Data":"89d175dd8675eb83a0af137a5789025fc89fd6eecced4aeec664f768661099bc"} Jul 06 23:35:02.641125 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:02.641086 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-6ph7p" event={"ID":"d948f999-2d98-4c3e-92fa-19cef82771ae","Type":"ContainerStarted","Data":"6ecff1f3e31695e2953db034f6a4f92c957b44d77a36b7cb19fd453f6d0926c6"} Jul 06 23:35:02.641564 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:02.641352 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-6ph7p" Jul 06 23:35:02.656664 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:02.656613 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-6ph7p" podStartSLOduration=1.282488018 podStartE2EDuration="2.656599314s" totalImagesPullingTime="1.374111296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:35:02.655922957 +0000 UTC m=+613.369977526" watchObservedRunningTime="2026-07-06 23:35:02.656599314 +0000 UTC m=+613.370653884" Jul 06 23:35:07.674101 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:07.674065 2571 generic.go:356] "Generic (PLEG): container finished" podID="d0c72526-90d2-46aa-ac1f-4bd0625ba248" containerID="59f180746e854fef636832b79ca0670f66fa6770fe7dd97421b9a30762f20650" exitCode=0 Jul 06 23:35:07.674519 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:07.674136 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" event={"ID":"d0c72526-90d2-46aa-ac1f-4bd0625ba248","Type":"ContainerDied","Data":"59f180746e854fef636832b79ca0670f66fa6770fe7dd97421b9a30762f20650"} Jul 06 23:35:07.674519 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:07.674513 2571 scope.go:117] "RemoveContainer" containerID="59f180746e854fef636832b79ca0670f66fa6770fe7dd97421b9a30762f20650" Jul 06 23:35:07.675507 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:07.675479 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:35:07.675587 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:07.675546 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:35:07.675622 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:07.675596 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:35:08.680959 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:08.680925 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" event={"ID":"d0c72526-90d2-46aa-ac1f-4bd0625ba248","Type":"ContainerStarted","Data":"30505b40f921ca66ce562cd69b6d5b14f8fb60b331e1c505e9f12bde948e9c05"} Jul 06 23:35:08.869343 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:08.869301 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:35:08.869609 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:08.869582 2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-24flx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 06 23:35:08.869703 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:08.869641 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" podUID="d0c72526-90d2-46aa-ac1f-4bd0625ba248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 06 23:35:09.869736 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:09.869697 2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-24flx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 06 23:35:09.870162 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:09.869780 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" podUID="d0c72526-90d2-46aa-ac1f-4bd0625ba248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 06 23:35:10.870362 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:10.870275 2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-24flx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 06 23:35:10.870362 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:10.870328 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" podUID="d0c72526-90d2-46aa-ac1f-4bd0625ba248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 06 23:35:11.869642 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:11.869607 2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-24flx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 06 23:35:11.869838 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:11.869665 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" podUID="d0c72526-90d2-46aa-ac1f-4bd0625ba248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 06 23:35:12.873386 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:12.873359 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:35:12.873805 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:12.873737 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:35:12.874519 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:12.874502 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-24flx" Jul 06 23:35:13.648613 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:13.648585 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-6ph7p" Jul 06 23:35:21.284231 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.284201 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c"] Jul 06 23:35:21.286472 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.286456 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.288798 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.288776 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 23:35:21.288926 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.288776 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 23:35:21.289577 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.289550 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 06 23:35:21.289698 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.289595 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-g2mcc\"" Jul 06 23:35:21.295602 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.295579 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c"] Jul 06 23:35:21.302587 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.302564 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkrtd\" (UniqueName: \"kubernetes.io/projected/d8c0c1a2-a64c-4430-9fc6-4091f680d85b-kube-api-access-gkrtd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c\" (UID: \"d8c0c1a2-a64c-4430-9fc6-4091f680d85b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.302748 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.302603 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c0c1a2-a64c-4430-9fc6-4091f680d85b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c\" (UID: \"d8c0c1a2-a64c-4430-9fc6-4091f680d85b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.302748 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.302633 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c0c1a2-a64c-4430-9fc6-4091f680d85b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c\" (UID: \"d8c0c1a2-a64c-4430-9fc6-4091f680d85b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.302748 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.302702 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c0c1a2-a64c-4430-9fc6-4091f680d85b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c\" (UID: \"d8c0c1a2-a64c-4430-9fc6-4091f680d85b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.302880 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.302795 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c0c1a2-a64c-4430-9fc6-4091f680d85b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c\" (UID: \"d8c0c1a2-a64c-4430-9fc6-4091f680d85b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.302880 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.302841 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c0c1a2-a64c-4430-9fc6-4091f680d85b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c\" (UID: \"d8c0c1a2-a64c-4430-9fc6-4091f680d85b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.403758 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.403699 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c0c1a2-a64c-4430-9fc6-4091f680d85b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c\" (UID: \"d8c0c1a2-a64c-4430-9fc6-4091f680d85b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.403946 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.403768 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c0c1a2-a64c-4430-9fc6-4091f680d85b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c\" (UID: \"d8c0c1a2-a64c-4430-9fc6-4091f680d85b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.403946 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.403826 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c0c1a2-a64c-4430-9fc6-4091f680d85b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c\" (UID: \"d8c0c1a2-a64c-4430-9fc6-4091f680d85b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.403946 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.403878 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c0c1a2-a64c-4430-9fc6-4091f680d85b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c\" (UID: \"d8c0c1a2-a64c-4430-9fc6-4091f680d85b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.403946 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.403912 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c0c1a2-a64c-4430-9fc6-4091f680d85b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c\" (UID: \"d8c0c1a2-a64c-4430-9fc6-4091f680d85b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.404177 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.403951 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkrtd\" (UniqueName: \"kubernetes.io/projected/d8c0c1a2-a64c-4430-9fc6-4091f680d85b-kube-api-access-gkrtd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c\" (UID: \"d8c0c1a2-a64c-4430-9fc6-4091f680d85b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.404277 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.404253 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c0c1a2-a64c-4430-9fc6-4091f680d85b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c\" (UID: \"d8c0c1a2-a64c-4430-9fc6-4091f680d85b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.404336 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.404263 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c0c1a2-a64c-4430-9fc6-4091f680d85b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c\" (UID: \"d8c0c1a2-a64c-4430-9fc6-4091f680d85b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.404373 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.404331 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c0c1a2-a64c-4430-9fc6-4091f680d85b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c\" (UID: \"d8c0c1a2-a64c-4430-9fc6-4091f680d85b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.406103 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.406084 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c0c1a2-a64c-4430-9fc6-4091f680d85b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c\" (UID: \"d8c0c1a2-a64c-4430-9fc6-4091f680d85b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.406367 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.406350 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c0c1a2-a64c-4430-9fc6-4091f680d85b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c\" (UID: \"d8c0c1a2-a64c-4430-9fc6-4091f680d85b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.411838 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.411813 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkrtd\" (UniqueName: \"kubernetes.io/projected/d8c0c1a2-a64c-4430-9fc6-4091f680d85b-kube-api-access-gkrtd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c\" (UID: \"d8c0c1a2-a64c-4430-9fc6-4091f680d85b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.599805 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.599706 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:21.743006 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.742979 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c"] Jul 06 23:35:21.744974 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:35:21.744942 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c0c1a2_a64c_4430_9fc6_4091f680d85b.slice/crio-fa215b407fbeb56d21679be35f38f94a7e843cc31b466f60fe730e8049b82dfc WatchSource:0}: Error finding container fa215b407fbeb56d21679be35f38f94a7e843cc31b466f60fe730e8049b82dfc: Status 404 returned error can't find the container with id fa215b407fbeb56d21679be35f38f94a7e843cc31b466f60fe730e8049b82dfc Jul 06 23:35:21.746561 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:21.746537 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 23:35:22.754176 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:22.754143 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" event={"ID":"d8c0c1a2-a64c-4430-9fc6-4091f680d85b","Type":"ContainerStarted","Data":"fa215b407fbeb56d21679be35f38f94a7e843cc31b466f60fe730e8049b82dfc"} Jul 06 23:35:23.760003 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:23.759965 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" event={"ID":"d8c0c1a2-a64c-4430-9fc6-4091f680d85b","Type":"ContainerStarted","Data":"24d8fbdbd2becbcbbc0943c247060385258af7467d899ce873e0b73d332a68c0"} Jul 06 23:35:23.779639 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:23.779504 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" podStartSLOduration=1.742896396 podStartE2EDuration="2.779484622s" totalImagesPullingTime="1.036588226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:35:23.779043338 +0000 UTC m=+634.493097907" watchObservedRunningTime="2026-07-06 23:35:23.779484622 +0000 UTC m=+634.493539194" Jul 06 23:35:31.600234 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:31.600193 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:31.600652 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:31.600285 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:31.612673 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:31.612650 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:31.812070 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:31.812043 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c" Jul 06 23:35:55.475004 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.474968 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq"] Jul 06 23:35:55.478741 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.478696 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.481139 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.481118 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 23:35:55.486534 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.486292 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq"] Jul 06 23:35:55.574863 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.574836 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq2hc\" (UniqueName: \"kubernetes.io/projected/eda0239d-0167-45e7-947c-5f0167a0dfe6-kube-api-access-vq2hc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq\" (UID: \"eda0239d-0167-45e7-947c-5f0167a0dfe6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.575029 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.574870 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eda0239d-0167-45e7-947c-5f0167a0dfe6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq\" (UID: \"eda0239d-0167-45e7-947c-5f0167a0dfe6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.575029 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.574926 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eda0239d-0167-45e7-947c-5f0167a0dfe6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq\" (UID: \"eda0239d-0167-45e7-947c-5f0167a0dfe6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.575029 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.574954 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eda0239d-0167-45e7-947c-5f0167a0dfe6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq\" (UID: \"eda0239d-0167-45e7-947c-5f0167a0dfe6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.575029 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.574974 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eda0239d-0167-45e7-947c-5f0167a0dfe6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq\" (UID: \"eda0239d-0167-45e7-947c-5f0167a0dfe6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.575220 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.575083 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eda0239d-0167-45e7-947c-5f0167a0dfe6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq\" (UID: \"eda0239d-0167-45e7-947c-5f0167a0dfe6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.676244 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.676211 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eda0239d-0167-45e7-947c-5f0167a0dfe6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq\" (UID: \"eda0239d-0167-45e7-947c-5f0167a0dfe6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.676386 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.676246 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eda0239d-0167-45e7-947c-5f0167a0dfe6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq\" (UID: \"eda0239d-0167-45e7-947c-5f0167a0dfe6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.676386 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.676285 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eda0239d-0167-45e7-947c-5f0167a0dfe6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq\" (UID: \"eda0239d-0167-45e7-947c-5f0167a0dfe6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.676386 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.676314 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq2hc\" (UniqueName: \"kubernetes.io/projected/eda0239d-0167-45e7-947c-5f0167a0dfe6-kube-api-access-vq2hc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq\" (UID: \"eda0239d-0167-45e7-947c-5f0167a0dfe6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.676386 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.676331 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eda0239d-0167-45e7-947c-5f0167a0dfe6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq\" (UID: \"eda0239d-0167-45e7-947c-5f0167a0dfe6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.676386 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.676357 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eda0239d-0167-45e7-947c-5f0167a0dfe6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq\" (UID: \"eda0239d-0167-45e7-947c-5f0167a0dfe6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.676842 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.676811 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eda0239d-0167-45e7-947c-5f0167a0dfe6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq\" (UID: \"eda0239d-0167-45e7-947c-5f0167a0dfe6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.676842 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.676832 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eda0239d-0167-45e7-947c-5f0167a0dfe6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq\" (UID: \"eda0239d-0167-45e7-947c-5f0167a0dfe6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.676989 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.676890 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eda0239d-0167-45e7-947c-5f0167a0dfe6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq\" (UID: \"eda0239d-0167-45e7-947c-5f0167a0dfe6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.678614 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.678594 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eda0239d-0167-45e7-947c-5f0167a0dfe6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq\" (UID: \"eda0239d-0167-45e7-947c-5f0167a0dfe6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.678880 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.678862 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eda0239d-0167-45e7-947c-5f0167a0dfe6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq\" (UID: \"eda0239d-0167-45e7-947c-5f0167a0dfe6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.684618 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.684596 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq2hc\" (UniqueName: \"kubernetes.io/projected/eda0239d-0167-45e7-947c-5f0167a0dfe6-kube-api-access-vq2hc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq\" (UID: \"eda0239d-0167-45e7-947c-5f0167a0dfe6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.790755 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.790661 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:35:55.919480 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:55.919459 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq"] Jul 06 23:35:55.921943 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:35:55.921915 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda0239d_0167_45e7_947c_5f0167a0dfe6.slice/crio-20f03916c45f32174bf2a963e0f4ed0c4c49ab47a88e1cc0ebaca6c65592d133 WatchSource:0}: Error finding container 20f03916c45f32174bf2a963e0f4ed0c4c49ab47a88e1cc0ebaca6c65592d133: Status 404 returned error can't find the container with id 20f03916c45f32174bf2a963e0f4ed0c4c49ab47a88e1cc0ebaca6c65592d133 Jul 06 23:35:56.923342 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:56.923300 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" event={"ID":"eda0239d-0167-45e7-947c-5f0167a0dfe6","Type":"ContainerStarted","Data":"49ea473e109e7963327f7114067996183693c653510c7a949e9b1c8cf881cb59"} Jul 06 23:35:56.923743 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:56.923365 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" event={"ID":"eda0239d-0167-45e7-947c-5f0167a0dfe6","Type":"ContainerStarted","Data":"20f03916c45f32174bf2a963e0f4ed0c4c49ab47a88e1cc0ebaca6c65592d133"} Jul 06 23:35:56.956890 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:35:56.956817 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" podStartSLOduration=1.777482291 podStartE2EDuration="1.956795806s" totalImagesPullingTime="179.313515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:35:56.94197897 +0000 UTC m=+667.656033537" watchObservedRunningTime="2026-07-06 23:35:56.956795806 +0000 UTC m=+667.670850378" Jul 06 23:36:05.791662 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:05.791631 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:36:05.792169 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:05.791708 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:36:05.804208 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:05.804186 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:36:05.980192 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:05.980166 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq" Jul 06 23:36:13.866992 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:13.866957 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7"] Jul 06 23:36:13.870438 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:13.870416 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:13.872794 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:13.872708 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 06 23:36:13.881087 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:13.881064 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7"] Jul 06 23:36:14.026365 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.026333 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/216e2e88-850e-4ea7-aaea-e47e44a425ff-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7\" (UID: \"216e2e88-850e-4ea7-aaea-e47e44a425ff\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.026365 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.026369 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216e2e88-850e-4ea7-aaea-e47e44a425ff-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7\" (UID: \"216e2e88-850e-4ea7-aaea-e47e44a425ff\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.026584 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.026391 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/216e2e88-850e-4ea7-aaea-e47e44a425ff-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7\" (UID: \"216e2e88-850e-4ea7-aaea-e47e44a425ff\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.026584 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.026419 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/216e2e88-850e-4ea7-aaea-e47e44a425ff-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7\" (UID: \"216e2e88-850e-4ea7-aaea-e47e44a425ff\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.026584 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.026452 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/216e2e88-850e-4ea7-aaea-e47e44a425ff-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7\" (UID: \"216e2e88-850e-4ea7-aaea-e47e44a425ff\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.026584 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.026508 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h9jn\" (UniqueName: \"kubernetes.io/projected/216e2e88-850e-4ea7-aaea-e47e44a425ff-kube-api-access-6h9jn\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7\" (UID: \"216e2e88-850e-4ea7-aaea-e47e44a425ff\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.127544 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.127450 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h9jn\" (UniqueName: \"kubernetes.io/projected/216e2e88-850e-4ea7-aaea-e47e44a425ff-kube-api-access-6h9jn\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7\" (UID: \"216e2e88-850e-4ea7-aaea-e47e44a425ff\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.127544 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.127537 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/216e2e88-850e-4ea7-aaea-e47e44a425ff-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7\" (UID: \"216e2e88-850e-4ea7-aaea-e47e44a425ff\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.127884 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.127575 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216e2e88-850e-4ea7-aaea-e47e44a425ff-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7\" (UID: \"216e2e88-850e-4ea7-aaea-e47e44a425ff\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.127884 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.127607 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/216e2e88-850e-4ea7-aaea-e47e44a425ff-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7\" (UID: \"216e2e88-850e-4ea7-aaea-e47e44a425ff\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.127884 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.127657 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/216e2e88-850e-4ea7-aaea-e47e44a425ff-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7\" (UID: \"216e2e88-850e-4ea7-aaea-e47e44a425ff\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.127884 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.127681 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/216e2e88-850e-4ea7-aaea-e47e44a425ff-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7\" (UID: \"216e2e88-850e-4ea7-aaea-e47e44a425ff\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.128074 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.128049 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216e2e88-850e-4ea7-aaea-e47e44a425ff-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7\" (UID: \"216e2e88-850e-4ea7-aaea-e47e44a425ff\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.128150 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.128124 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/216e2e88-850e-4ea7-aaea-e47e44a425ff-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7\" (UID: \"216e2e88-850e-4ea7-aaea-e47e44a425ff\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.128229 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.128206 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/216e2e88-850e-4ea7-aaea-e47e44a425ff-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7\" (UID: \"216e2e88-850e-4ea7-aaea-e47e44a425ff\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.130121 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.130100 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/216e2e88-850e-4ea7-aaea-e47e44a425ff-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7\" (UID: \"216e2e88-850e-4ea7-aaea-e47e44a425ff\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.130323 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.130301 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/216e2e88-850e-4ea7-aaea-e47e44a425ff-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7\" (UID: \"216e2e88-850e-4ea7-aaea-e47e44a425ff\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.139049 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.139023 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h9jn\" (UniqueName: \"kubernetes.io/projected/216e2e88-850e-4ea7-aaea-e47e44a425ff-kube-api-access-6h9jn\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7\" (UID: \"216e2e88-850e-4ea7-aaea-e47e44a425ff\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.182035 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.181998 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:14.318629 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:14.318568 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7"] Jul 06 23:36:14.320793 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:36:14.320765 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod216e2e88_850e_4ea7_aaea_e47e44a425ff.slice/crio-84ffa0ab92dd503dc55e4a5b26d1443fec24d8f7086b458b1a740595733d11b2 WatchSource:0}: Error finding container 84ffa0ab92dd503dc55e4a5b26d1443fec24d8f7086b458b1a740595733d11b2: Status 404 returned error can't find the container with id 84ffa0ab92dd503dc55e4a5b26d1443fec24d8f7086b458b1a740595733d11b2 Jul 06 23:36:15.016417 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:15.016384 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" event={"ID":"216e2e88-850e-4ea7-aaea-e47e44a425ff","Type":"ContainerStarted","Data":"de230200ad879e728bc415d802a6b92fdc568ff393998180b88a096407a3198b"} Jul 06 23:36:15.016417 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:15.016422 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" event={"ID":"216e2e88-850e-4ea7-aaea-e47e44a425ff","Type":"ContainerStarted","Data":"84ffa0ab92dd503dc55e4a5b26d1443fec24d8f7086b458b1a740595733d11b2"} Jul 06 23:36:15.035346 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:15.035294 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" podStartSLOduration=1.87019756 podStartE2EDuration="2.035281578s" totalImagesPullingTime="165.084018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:36:15.035040004 +0000 UTC m=+685.749094574" watchObservedRunningTime="2026-07-06 23:36:15.035281578 +0000 UTC m=+685.749336151" Jul 06 23:36:24.182816 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:24.182781 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:24.183201 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:24.182833 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:24.195177 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:24.195155 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:25.086864 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:25.086833 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7" Jul 06 23:36:57.231501 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:57.231420 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-85f77bb897-8zv69"] Jul 06 23:36:57.234908 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:57.234890 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85f77bb897-8zv69" Jul 06 23:36:57.242699 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:57.242675 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85f77bb897-8zv69"] Jul 06 23:36:57.285045 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:57.285000 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdrj4\" (UniqueName: \"kubernetes.io/projected/bf0205d9-de1a-4043-b6a4-68576017e352-kube-api-access-mdrj4\") pod \"authorino-85f77bb897-8zv69\" (UID: \"bf0205d9-de1a-4043-b6a4-68576017e352\") " pod="kuadrant-system/authorino-85f77bb897-8zv69" Jul 06 23:36:57.285229 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:57.285071 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf0205d9-de1a-4043-b6a4-68576017e352-tls-cert\") pod \"authorino-85f77bb897-8zv69\" (UID: \"bf0205d9-de1a-4043-b6a4-68576017e352\") " pod="kuadrant-system/authorino-85f77bb897-8zv69" Jul 06 23:36:57.386241 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:57.386207 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdrj4\" (UniqueName: \"kubernetes.io/projected/bf0205d9-de1a-4043-b6a4-68576017e352-kube-api-access-mdrj4\") pod \"authorino-85f77bb897-8zv69\" (UID: \"bf0205d9-de1a-4043-b6a4-68576017e352\") " pod="kuadrant-system/authorino-85f77bb897-8zv69" Jul 06 23:36:57.386430 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:57.386263 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf0205d9-de1a-4043-b6a4-68576017e352-tls-cert\") pod \"authorino-85f77bb897-8zv69\" (UID: \"bf0205d9-de1a-4043-b6a4-68576017e352\") " pod="kuadrant-system/authorino-85f77bb897-8zv69" Jul 06 23:36:57.388785 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:57.388765 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf0205d9-de1a-4043-b6a4-68576017e352-tls-cert\") pod \"authorino-85f77bb897-8zv69\" (UID: \"bf0205d9-de1a-4043-b6a4-68576017e352\") " pod="kuadrant-system/authorino-85f77bb897-8zv69" Jul 06 23:36:57.396228 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:57.396196 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdrj4\" (UniqueName: \"kubernetes.io/projected/bf0205d9-de1a-4043-b6a4-68576017e352-kube-api-access-mdrj4\") pod \"authorino-85f77bb897-8zv69\" (UID: \"bf0205d9-de1a-4043-b6a4-68576017e352\") " pod="kuadrant-system/authorino-85f77bb897-8zv69" Jul 06 23:36:57.545221 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:57.545149 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85f77bb897-8zv69" Jul 06 23:36:57.686535 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:57.686507 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85f77bb897-8zv69"] Jul 06 23:36:57.689235 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:36:57.689209 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf0205d9_de1a_4043_b6a4_68576017e352.slice/crio-f4a0f57b8070719b24d6b2f4e6723e5c0e648b537e5a1828a53f32e5dd7f5595 WatchSource:0}: Error finding container f4a0f57b8070719b24d6b2f4e6723e5c0e648b537e5a1828a53f32e5dd7f5595: Status 404 returned error can't find the container with id f4a0f57b8070719b24d6b2f4e6723e5c0e648b537e5a1828a53f32e5dd7f5595 Jul 06 23:36:58.258218 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:58.258179 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85f77bb897-8zv69" event={"ID":"bf0205d9-de1a-4043-b6a4-68576017e352","Type":"ContainerStarted","Data":"27e609519f63e563585d9a4bcbf893cc65a0b5e8c716ba7818de9c2419836762"} Jul 06 23:36:58.258218 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:58.258222 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85f77bb897-8zv69" event={"ID":"bf0205d9-de1a-4043-b6a4-68576017e352","Type":"ContainerStarted","Data":"f4a0f57b8070719b24d6b2f4e6723e5c0e648b537e5a1828a53f32e5dd7f5595"} Jul 06 23:36:58.283339 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:58.283229 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-85f77bb897-8zv69" podStartSLOduration=0.946554812 podStartE2EDuration="1.283204486s" totalImagesPullingTime="336.649674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:36:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:36:58.281580597 +0000 UTC m=+728.995635168" watchObservedRunningTime="2026-07-06 23:36:58.283204486 +0000 UTC m=+728.997259058" Jul 06 23:36:58.317762 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:58.316459 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f6cbb5bdd-k8drl"] Jul 06 23:36:58.317762 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:58.317050 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-7f6cbb5bdd-k8drl" podUID="7e68140a-2fb6-406d-a185-0ba44acccf50" containerName="authorino" containerID="cri-o://55b907224719956f7e5b1478f4f6a62b835f6dae31d02c92fda191f6d80e7a3a" gracePeriod=30 Jul 06 23:36:58.590661 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:58.590627 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f6cbb5bdd-k8drl" Jul 06 23:36:58.697751 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:58.697705 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnpjl\" (UniqueName: \"kubernetes.io/projected/7e68140a-2fb6-406d-a185-0ba44acccf50-kube-api-access-pnpjl\") pod \"7e68140a-2fb6-406d-a185-0ba44acccf50\" (UID: \"7e68140a-2fb6-406d-a185-0ba44acccf50\") " Jul 06 23:36:58.697931 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:58.697764 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e68140a-2fb6-406d-a185-0ba44acccf50-tls-cert\") pod \"7e68140a-2fb6-406d-a185-0ba44acccf50\" (UID: \"7e68140a-2fb6-406d-a185-0ba44acccf50\") " Jul 06 23:36:58.700029 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:58.699986 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e68140a-2fb6-406d-a185-0ba44acccf50-kube-api-access-pnpjl" (OuterVolumeSpecName: "kube-api-access-pnpjl") pod "7e68140a-2fb6-406d-a185-0ba44acccf50" (UID: "7e68140a-2fb6-406d-a185-0ba44acccf50"). InnerVolumeSpecName "kube-api-access-pnpjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:36:58.707800 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:58.707771 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e68140a-2fb6-406d-a185-0ba44acccf50-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "7e68140a-2fb6-406d-a185-0ba44acccf50" (UID: "7e68140a-2fb6-406d-a185-0ba44acccf50"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:36:58.798582 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:58.798550 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnpjl\" (UniqueName: \"kubernetes.io/projected/7e68140a-2fb6-406d-a185-0ba44acccf50-kube-api-access-pnpjl\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:36:58.798582 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:58.798579 2571 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e68140a-2fb6-406d-a185-0ba44acccf50-tls-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:36:59.264895 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:59.264856 2571 generic.go:356] "Generic (PLEG): container finished" podID="7e68140a-2fb6-406d-a185-0ba44acccf50" containerID="55b907224719956f7e5b1478f4f6a62b835f6dae31d02c92fda191f6d80e7a3a" exitCode=0 Jul 06 23:36:59.265303 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:59.264905 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f6cbb5bdd-k8drl" Jul 06 23:36:59.265303 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:59.264940 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f6cbb5bdd-k8drl" event={"ID":"7e68140a-2fb6-406d-a185-0ba44acccf50","Type":"ContainerDied","Data":"55b907224719956f7e5b1478f4f6a62b835f6dae31d02c92fda191f6d80e7a3a"} Jul 06 23:36:59.265303 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:59.264983 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f6cbb5bdd-k8drl" event={"ID":"7e68140a-2fb6-406d-a185-0ba44acccf50","Type":"ContainerDied","Data":"8f5bde8b61557a1b2714a71da7f1eb1b31bf6bedc7fca98ca95659891f90b249"} Jul 06 23:36:59.265303 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:59.265003 2571 scope.go:117] "RemoveContainer" containerID="55b907224719956f7e5b1478f4f6a62b835f6dae31d02c92fda191f6d80e7a3a" Jul 06 23:36:59.274841 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:59.274818 2571 scope.go:117] "RemoveContainer" containerID="55b907224719956f7e5b1478f4f6a62b835f6dae31d02c92fda191f6d80e7a3a" Jul 06 23:36:59.275225 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:36:59.275194 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b907224719956f7e5b1478f4f6a62b835f6dae31d02c92fda191f6d80e7a3a\": container with ID starting with 55b907224719956f7e5b1478f4f6a62b835f6dae31d02c92fda191f6d80e7a3a not found: ID does not exist" containerID="55b907224719956f7e5b1478f4f6a62b835f6dae31d02c92fda191f6d80e7a3a" Jul 06 23:36:59.275345 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:59.275234 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b907224719956f7e5b1478f4f6a62b835f6dae31d02c92fda191f6d80e7a3a"} err="failed to get container status \"55b907224719956f7e5b1478f4f6a62b835f6dae31d02c92fda191f6d80e7a3a\": rpc error: code = NotFound desc = could not find container \"55b907224719956f7e5b1478f4f6a62b835f6dae31d02c92fda191f6d80e7a3a\": container with ID starting with 55b907224719956f7e5b1478f4f6a62b835f6dae31d02c92fda191f6d80e7a3a not found: ID does not exist" Jul 06 23:36:59.297923 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:59.297889 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f6cbb5bdd-k8drl"] Jul 06 23:36:59.304085 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:59.304057 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7f6cbb5bdd-k8drl"] Jul 06 23:36:59.786405 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:36:59.786370 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e68140a-2fb6-406d-a185-0ba44acccf50" path="/var/lib/kubelet/pods/7e68140a-2fb6-406d-a185-0ba44acccf50/volumes" Jul 06 23:38:13.590121 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:13.590037 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-585888c566-5s5m7"] Jul 06 23:38:13.590643 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:13.590424 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e68140a-2fb6-406d-a185-0ba44acccf50" containerName="authorino" Jul 06 23:38:13.590643 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:13.590487 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e68140a-2fb6-406d-a185-0ba44acccf50" containerName="authorino" Jul 06 23:38:13.590643 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:13.590608 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e68140a-2fb6-406d-a185-0ba44acccf50" containerName="authorino" Jul 06 23:38:13.594854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:13.594836 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-585888c566-5s5m7" Jul 06 23:38:13.598267 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:13.598245 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 23:38:13.598391 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:13.598246 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mfnr2\"" Jul 06 23:38:13.598391 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:13.598286 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 23:38:13.605300 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:13.605271 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-585888c566-5s5m7"] Jul 06 23:38:13.739359 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:13.739316 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndqm6\" (UniqueName: \"kubernetes.io/projected/80f2e998-37f7-4e22-a3c2-e4158bd937e8-kube-api-access-ndqm6\") pod \"maas-controller-585888c566-5s5m7\" (UID: \"80f2e998-37f7-4e22-a3c2-e4158bd937e8\") " pod="opendatahub/maas-controller-585888c566-5s5m7" Jul 06 23:38:13.739359 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:13.739364 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80f2e998-37f7-4e22-a3c2-e4158bd937e8-webhook-cert\") pod \"maas-controller-585888c566-5s5m7\" (UID: \"80f2e998-37f7-4e22-a3c2-e4158bd937e8\") " pod="opendatahub/maas-controller-585888c566-5s5m7" Jul 06 23:38:13.840289 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:13.840206 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndqm6\" (UniqueName: \"kubernetes.io/projected/80f2e998-37f7-4e22-a3c2-e4158bd937e8-kube-api-access-ndqm6\") pod \"maas-controller-585888c566-5s5m7\" (UID: \"80f2e998-37f7-4e22-a3c2-e4158bd937e8\") " pod="opendatahub/maas-controller-585888c566-5s5m7" Jul 06 23:38:13.840289 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:13.840252 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80f2e998-37f7-4e22-a3c2-e4158bd937e8-webhook-cert\") pod \"maas-controller-585888c566-5s5m7\" (UID: \"80f2e998-37f7-4e22-a3c2-e4158bd937e8\") " pod="opendatahub/maas-controller-585888c566-5s5m7" Jul 06 23:38:13.842713 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:13.842691 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80f2e998-37f7-4e22-a3c2-e4158bd937e8-webhook-cert\") pod \"maas-controller-585888c566-5s5m7\" (UID: \"80f2e998-37f7-4e22-a3c2-e4158bd937e8\") " pod="opendatahub/maas-controller-585888c566-5s5m7" Jul 06 23:38:13.849463 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:13.849441 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndqm6\" (UniqueName: \"kubernetes.io/projected/80f2e998-37f7-4e22-a3c2-e4158bd937e8-kube-api-access-ndqm6\") pod \"maas-controller-585888c566-5s5m7\" (UID: \"80f2e998-37f7-4e22-a3c2-e4158bd937e8\") " pod="opendatahub/maas-controller-585888c566-5s5m7" Jul 06 23:38:13.906028 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:13.906005 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-585888c566-5s5m7" Jul 06 23:38:14.037149 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:14.037124 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-585888c566-5s5m7"] Jul 06 23:38:14.039001 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:38:14.038976 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f2e998_37f7_4e22_a3c2_e4158bd937e8.slice/crio-80f6a13b899aaed5b3cc4e3dca66fe484056d69c321c0410b09ac2ccd342038d WatchSource:0}: Error finding container 80f6a13b899aaed5b3cc4e3dca66fe484056d69c321c0410b09ac2ccd342038d: Status 404 returned error can't find the container with id 80f6a13b899aaed5b3cc4e3dca66fe484056d69c321c0410b09ac2ccd342038d Jul 06 23:38:14.657355 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:14.657321 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-585888c566-5s5m7" event={"ID":"80f2e998-37f7-4e22-a3c2-e4158bd937e8","Type":"ContainerStarted","Data":"36aebea191745a94a0acb606783f6815116911b135ea38115bc109eb452685f3"} Jul 06 23:38:14.657896 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:14.657372 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-585888c566-5s5m7" event={"ID":"80f2e998-37f7-4e22-a3c2-e4158bd937e8","Type":"ContainerStarted","Data":"80f6a13b899aaed5b3cc4e3dca66fe484056d69c321c0410b09ac2ccd342038d"} Jul 06 23:38:14.657896 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:14.657414 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-585888c566-5s5m7" Jul 06 23:38:14.674177 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:14.674129 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-585888c566-5s5m7" podStartSLOduration=1.192294707 podStartE2EDuration="1.674109568s" totalImagesPullingTime="481.814861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:38:14.672620285 +0000 UTC m=+805.386674856" watchObservedRunningTime="2026-07-06 23:38:14.674109568 +0000 UTC m=+805.388164128" Jul 06 23:38:25.667872 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:38:25.667840 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-585888c566-5s5m7" Jul 06 23:39:49.725307 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:39:49.725237 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 06 23:39:49.725307 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:39:49.725251 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 06 23:43:00.258120 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:00.258021 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b"] Jul 06 23:43:00.260853 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:00.258294 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b" podUID="7955cb15-e2d6-4a22-8013-e97bb65c831d" containerName="manager" containerID="cri-o://779e6a01aba1792748ff14127adb2bb7aa98d93db853846b50a6a04dfd5bc228" gracePeriod=10 Jul 06 23:43:02.911687 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:02.911660 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b" Jul 06 23:43:02.956672 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:02.956640 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7955cb15-e2d6-4a22-8013-e97bb65c831d-extensions-socket-volume\") pod \"7955cb15-e2d6-4a22-8013-e97bb65c831d\" (UID: \"7955cb15-e2d6-4a22-8013-e97bb65c831d\") " Jul 06 23:43:02.956847 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:02.956679 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cvxg\" (UniqueName: \"kubernetes.io/projected/7955cb15-e2d6-4a22-8013-e97bb65c831d-kube-api-access-4cvxg\") pod \"7955cb15-e2d6-4a22-8013-e97bb65c831d\" (UID: \"7955cb15-e2d6-4a22-8013-e97bb65c831d\") " Jul 06 23:43:02.957100 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:02.957079 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7955cb15-e2d6-4a22-8013-e97bb65c831d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7955cb15-e2d6-4a22-8013-e97bb65c831d" (UID: "7955cb15-e2d6-4a22-8013-e97bb65c831d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:43:02.959065 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:02.959038 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7955cb15-e2d6-4a22-8013-e97bb65c831d-kube-api-access-4cvxg" (OuterVolumeSpecName: "kube-api-access-4cvxg") pod "7955cb15-e2d6-4a22-8013-e97bb65c831d" (UID: "7955cb15-e2d6-4a22-8013-e97bb65c831d"). InnerVolumeSpecName "kube-api-access-4cvxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:43:03.057629 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:03.057555 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7955cb15-e2d6-4a22-8013-e97bb65c831d-extensions-socket-volume\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:43:03.057629 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:03.057581 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cvxg\" (UniqueName: \"kubernetes.io/projected/7955cb15-e2d6-4a22-8013-e97bb65c831d-kube-api-access-4cvxg\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:43:03.182139 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:03.182105 2571 generic.go:356] "Generic (PLEG): container finished" podID="7955cb15-e2d6-4a22-8013-e97bb65c831d" containerID="779e6a01aba1792748ff14127adb2bb7aa98d93db853846b50a6a04dfd5bc228" exitCode=0 Jul 06 23:43:03.182297 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:03.182148 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b" Jul 06 23:43:03.182297 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:03.182170 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b" event={"ID":"7955cb15-e2d6-4a22-8013-e97bb65c831d","Type":"ContainerDied","Data":"779e6a01aba1792748ff14127adb2bb7aa98d93db853846b50a6a04dfd5bc228"} Jul 06 23:43:03.182297 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:03.182222 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b" event={"ID":"7955cb15-e2d6-4a22-8013-e97bb65c831d","Type":"ContainerDied","Data":"66f18783ceede868d0b1893d5388cdf05342f66d4e454ad4dd1e812ac725f3fa"} Jul 06 23:43:03.182297 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:03.182251 2571 scope.go:117] "RemoveContainer" containerID="779e6a01aba1792748ff14127adb2bb7aa98d93db853846b50a6a04dfd5bc228" Jul 06 23:43:03.196053 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:03.196032 2571 scope.go:117] "RemoveContainer" containerID="779e6a01aba1792748ff14127adb2bb7aa98d93db853846b50a6a04dfd5bc228" Jul 06 23:43:03.196300 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:43:03.196278 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779e6a01aba1792748ff14127adb2bb7aa98d93db853846b50a6a04dfd5bc228\": container with ID starting with 779e6a01aba1792748ff14127adb2bb7aa98d93db853846b50a6a04dfd5bc228 not found: ID does not exist" containerID="779e6a01aba1792748ff14127adb2bb7aa98d93db853846b50a6a04dfd5bc228" Jul 06 23:43:03.196360 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:03.196312 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779e6a01aba1792748ff14127adb2bb7aa98d93db853846b50a6a04dfd5bc228"} err="failed to get container status \"779e6a01aba1792748ff14127adb2bb7aa98d93db853846b50a6a04dfd5bc228\": rpc error: code = NotFound desc = could not find container \"779e6a01aba1792748ff14127adb2bb7aa98d93db853846b50a6a04dfd5bc228\": container with ID starting with 779e6a01aba1792748ff14127adb2bb7aa98d93db853846b50a6a04dfd5bc228 not found: ID does not exist" Jul 06 23:43:03.205345 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:03.205323 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b"] Jul 06 23:43:03.212776 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:03.212756 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq9b"] Jul 06 23:43:03.786176 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:43:03.786140 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7955cb15-e2d6-4a22-8013-e97bb65c831d" path="/var/lib/kubelet/pods/7955cb15-e2d6-4a22-8013-e97bb65c831d/volumes" Jul 06 23:44:01.512315 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:01.512263 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rb7gf"] Jul 06 23:44:01.512855 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:01.512820 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7955cb15-e2d6-4a22-8013-e97bb65c831d" containerName="manager" Jul 06 23:44:01.512855 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:01.512840 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7955cb15-e2d6-4a22-8013-e97bb65c831d" containerName="manager" Jul 06 23:44:01.512978 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:01.512949 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7955cb15-e2d6-4a22-8013-e97bb65c831d" containerName="manager" Jul 06 23:44:01.517092 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:01.517069 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rb7gf" Jul 06 23:44:01.520755 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:01.520712 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-w7r4x\"" Jul 06 23:44:01.542658 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:01.542631 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rb7gf"] Jul 06 23:44:01.578516 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:01.578492 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h2ch\" (UniqueName: \"kubernetes.io/projected/33306557-1799-46c7-897d-ceae9e249f79-kube-api-access-5h2ch\") pod \"kuadrant-operator-controller-manager-86878c769f-rb7gf\" (UID: \"33306557-1799-46c7-897d-ceae9e249f79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rb7gf" Jul 06 23:44:01.578623 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:01.578520 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33306557-1799-46c7-897d-ceae9e249f79-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rb7gf\" (UID: \"33306557-1799-46c7-897d-ceae9e249f79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rb7gf" Jul 06 23:44:01.679627 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:01.679601 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2ch\" (UniqueName: \"kubernetes.io/projected/33306557-1799-46c7-897d-ceae9e249f79-kube-api-access-5h2ch\") pod \"kuadrant-operator-controller-manager-86878c769f-rb7gf\" (UID: \"33306557-1799-46c7-897d-ceae9e249f79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rb7gf" Jul 06 23:44:01.679744 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:01.679632 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33306557-1799-46c7-897d-ceae9e249f79-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rb7gf\" (UID: \"33306557-1799-46c7-897d-ceae9e249f79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rb7gf" Jul 06 23:44:01.680018 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:01.680000 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33306557-1799-46c7-897d-ceae9e249f79-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rb7gf\" (UID: \"33306557-1799-46c7-897d-ceae9e249f79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rb7gf" Jul 06 23:44:01.694929 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:01.694908 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h2ch\" (UniqueName: \"kubernetes.io/projected/33306557-1799-46c7-897d-ceae9e249f79-kube-api-access-5h2ch\") pod \"kuadrant-operator-controller-manager-86878c769f-rb7gf\" (UID: \"33306557-1799-46c7-897d-ceae9e249f79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rb7gf" Jul 06 23:44:01.826636 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:01.826566 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rb7gf" Jul 06 23:44:01.965352 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:01.965329 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rb7gf"] Jul 06 23:44:01.967799 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:44:01.967767 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33306557_1799_46c7_897d_ceae9e249f79.slice/crio-95afda069f1f2a8144b02125362a2088371a9e2467de64acf64e14372e422903 WatchSource:0}: Error finding container 95afda069f1f2a8144b02125362a2088371a9e2467de64acf64e14372e422903: Status 404 returned error can't find the container with id 95afda069f1f2a8144b02125362a2088371a9e2467de64acf64e14372e422903 Jul 06 23:44:02.486958 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:02.486918 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rb7gf" event={"ID":"33306557-1799-46c7-897d-ceae9e249f79","Type":"ContainerStarted","Data":"60d9c5d5d231d05313319415f2d3ba153c0adebb71dc3d0dbc8a160d81918d48"} Jul 06 23:44:02.487146 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:02.486979 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rb7gf" event={"ID":"33306557-1799-46c7-897d-ceae9e249f79","Type":"ContainerStarted","Data":"95afda069f1f2a8144b02125362a2088371a9e2467de64acf64e14372e422903"} Jul 06 23:44:02.487146 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:02.487009 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rb7gf" Jul 06 23:44:02.532851 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:02.532800 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rb7gf" podStartSLOduration=1.532785948 podStartE2EDuration="1.532785948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:44:02.520777413 +0000 UTC m=+1153.234831983" watchObservedRunningTime="2026-07-06 23:44:02.532785948 +0000 UTC m=+1153.246840519" Jul 06 23:44:13.497242 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:13.497209 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rb7gf" Jul 06 23:44:49.769858 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:49.769828 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 06 23:44:49.775139 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:44:49.775113 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 06 23:49:49.804131 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:49:49.804106 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 06 23:49:49.824654 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:49:49.824635 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 06 23:50:20.914827 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:20.914747 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5"] Jul 06 23:50:20.918826 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:20.918807 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:20.921387 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:20.921365 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e6a06da5-openshift-default-dockercfg-hq842\"" Jul 06 23:50:20.938153 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:20.938129 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5"] Jul 06 23:50:21.060624 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.060589 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-envoy\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.060624 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.060624 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-podinfo\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.060889 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.060643 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-token\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.060889 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.060740 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-workload-certs\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.060889 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.060787 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istiod-ca-cert\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.060889 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.060837 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-data\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.060889 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.060853 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gcwb\" (UniqueName: \"kubernetes.io/projected/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-kube-api-access-2gcwb\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.061049 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.060935 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-credential-socket\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.061049 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.060972 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-workload-socket\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.162077 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.162043 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-credential-socket\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.162271 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.162096 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-workload-socket\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.162271 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.162113 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-envoy\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.162271 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.162136 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-podinfo\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.162271 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.162158 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-token\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.162271 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.162207 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-workload-certs\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.162271 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.162242 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istiod-ca-cert\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.162580 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.162297 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-data\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.162580 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.162313 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gcwb\" (UniqueName: \"kubernetes.io/projected/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-kube-api-access-2gcwb\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.162685 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.162579 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-workload-socket\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.162685 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.162647 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-workload-certs\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.162818 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.162705 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-data\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.162818 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.162781 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-credential-socket\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.163079 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.163017 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istiod-ca-cert\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.164669 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.164649 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-envoy\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.164938 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.164893 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-podinfo\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.171066 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.171038 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-token\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.173155 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.173138 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gcwb\" (UniqueName: \"kubernetes.io/projected/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-kube-api-access-2gcwb\") pod \"e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.229358 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.229329 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:21.371378 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.371325 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5"] Jul 06 23:50:21.374333 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:50:21.374305 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bdd578e_f2f4_4182_b0cc_0078aa98e2d1.slice/crio-c12f320c158176ceb9bf3f9b58e5c10535c6dc4530fe291bf525a7a95e79e5b0 WatchSource:0}: Error finding container c12f320c158176ceb9bf3f9b58e5c10535c6dc4530fe291bf525a7a95e79e5b0: Status 404 returned error can't find the container with id c12f320c158176ceb9bf3f9b58e5c10535c6dc4530fe291bf525a7a95e79e5b0 Jul 06 23:50:21.376638 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.376606 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:50:21.376712 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.376670 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:50:21.376712 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.376700 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:50:21.446295 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:21.446266 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" event={"ID":"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1","Type":"ContainerStarted","Data":"c12f320c158176ceb9bf3f9b58e5c10535c6dc4530fe291bf525a7a95e79e5b0"} Jul 06 23:50:22.453965 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:22.453929 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" event={"ID":"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1","Type":"ContainerStarted","Data":"fed04a17b7bc529c7a60927f30fcd00e5611cd41249b5d82e840458cb54471f8"} Jul 06 23:50:22.478346 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:22.478292 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" podStartSLOduration=2.478271581 podStartE2EDuration="2.478271581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:22.477254322 +0000 UTC m=+1533.191308893" watchObservedRunningTime="2026-07-06 23:50:22.478271581 +0000 UTC m=+1533.192326153" Jul 06 23:50:23.230387 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:23.230356 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:23.235857 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:23.235835 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:23.465191 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:23.465149 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:23.465998 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:23.465979 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:27.401899 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:27.401869 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5"] Jul 06 23:50:27.402294 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:27.402067 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" podUID="3bdd578e-f2f4-4182-b0cc-0078aa98e2d1" containerName="istio-proxy" containerID="cri-o://fed04a17b7bc529c7a60927f30fcd00e5611cd41249b5d82e840458cb54471f8" gracePeriod=30 Jul 06 23:50:33.523903 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.523863 2571 generic.go:356] "Generic (PLEG): container finished" podID="3bdd578e-f2f4-4182-b0cc-0078aa98e2d1" containerID="fed04a17b7bc529c7a60927f30fcd00e5611cd41249b5d82e840458cb54471f8" exitCode=0 Jul 06 23:50:33.524277 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.523937 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" event={"ID":"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1","Type":"ContainerDied","Data":"fed04a17b7bc529c7a60927f30fcd00e5611cd41249b5d82e840458cb54471f8"} Jul 06 23:50:33.554621 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.554597 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:33.575132 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.575069 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istiod-ca-cert\") pod \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " Jul 06 23:50:33.575132 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.575107 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-envoy\") pod \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " Jul 06 23:50:33.575323 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.575140 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-workload-socket\") pod \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " Jul 06 23:50:33.575323 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.575180 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-token\") pod \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " Jul 06 23:50:33.575323 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.575206 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-podinfo\") pod \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " Jul 06 23:50:33.575323 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.575225 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gcwb\" (UniqueName: \"kubernetes.io/projected/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-kube-api-access-2gcwb\") pod \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " Jul 06 23:50:33.575323 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.575258 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-workload-certs\") pod \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " Jul 06 23:50:33.575323 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.575276 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-credential-socket\") pod \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " Jul 06 23:50:33.575323 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.575308 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-data\") pod \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\" (UID: \"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1\") " Jul 06 23:50:33.575670 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.575548 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3bdd578e-f2f4-4182-b0cc-0078aa98e2d1" (UID: "3bdd578e-f2f4-4182-b0cc-0078aa98e2d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:50:33.575670 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.575565 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3bdd578e-f2f4-4182-b0cc-0078aa98e2d1" (UID: "3bdd578e-f2f4-4182-b0cc-0078aa98e2d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:50:33.575945 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.575887 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3bdd578e-f2f4-4182-b0cc-0078aa98e2d1" (UID: "3bdd578e-f2f4-4182-b0cc-0078aa98e2d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:50:33.575945 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.575922 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3bdd578e-f2f4-4182-b0cc-0078aa98e2d1" (UID: "3bdd578e-f2f4-4182-b0cc-0078aa98e2d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:50:33.576219 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.576197 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-data" (OuterVolumeSpecName: "istio-data") pod "3bdd578e-f2f4-4182-b0cc-0078aa98e2d1" (UID: "3bdd578e-f2f4-4182-b0cc-0078aa98e2d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:50:33.578220 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.578193 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3bdd578e-f2f4-4182-b0cc-0078aa98e2d1" (UID: "3bdd578e-f2f4-4182-b0cc-0078aa98e2d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:50:33.578412 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.578224 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-kube-api-access-2gcwb" (OuterVolumeSpecName: "kube-api-access-2gcwb") pod "3bdd578e-f2f4-4182-b0cc-0078aa98e2d1" (UID: "3bdd578e-f2f4-4182-b0cc-0078aa98e2d1"). InnerVolumeSpecName "kube-api-access-2gcwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:50:33.578412 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.578346 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-token" (OuterVolumeSpecName: "istio-token") pod "3bdd578e-f2f4-4182-b0cc-0078aa98e2d1" (UID: "3bdd578e-f2f4-4182-b0cc-0078aa98e2d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:50:33.578412 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.578360 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3bdd578e-f2f4-4182-b0cc-0078aa98e2d1" (UID: "3bdd578e-f2f4-4182-b0cc-0078aa98e2d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 23:50:33.676987 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.676953 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-token\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:50:33.676987 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.676981 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-podinfo\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:50:33.676987 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.676991 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gcwb\" (UniqueName: \"kubernetes.io/projected/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-kube-api-access-2gcwb\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:50:33.677212 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.677002 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-workload-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:50:33.677212 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.677011 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-credential-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:50:33.677212 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.677019 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-data\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:50:33.677212 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.677027 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istiod-ca-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:50:33.677212 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.677034 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-istio-envoy\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:50:33.677212 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.677041 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1-workload-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:50:33.823110 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.823079 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc"] Jul 06 23:50:33.823431 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.823419 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bdd578e-f2f4-4182-b0cc-0078aa98e2d1" containerName="istio-proxy" Jul 06 23:50:33.823508 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.823433 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bdd578e-f2f4-4182-b0cc-0078aa98e2d1" containerName="istio-proxy" Jul 06 23:50:33.823508 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.823483 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bdd578e-f2f4-4182-b0cc-0078aa98e2d1" containerName="istio-proxy" Jul 06 23:50:33.825861 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.825840 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.828838 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.828817 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-5e8d5579-gw-openshift-default-dockercfg-s64sh\"" Jul 06 23:50:33.857080 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.857054 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc"] Jul 06 23:50:33.878412 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.878384 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-credential-socket\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.878551 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.878419 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-workload-certs\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.878551 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.878446 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-envoy\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.878551 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.878465 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bptmz\" (UniqueName: \"kubernetes.io/projected/8234dfb9-e2f4-48bd-b65c-882694d24da6-kube-api-access-bptmz\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.878551 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.878519 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-data\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.878551 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.878536 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-token\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.878733 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.878584 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-workload-socket\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.878733 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.878649 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8234dfb9-e2f4-48bd-b65c-882694d24da6-istiod-ca-cert\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.878733 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.878687 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-podinfo\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.979380 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.979282 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-workload-certs\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.979380 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.979326 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-envoy\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.979380 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.979345 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bptmz\" (UniqueName: \"kubernetes.io/projected/8234dfb9-e2f4-48bd-b65c-882694d24da6-kube-api-access-bptmz\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.979380 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.979378 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-data\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.979769 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.979404 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-token\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.979769 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.979426 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-workload-socket\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.979769 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.979462 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8234dfb9-e2f4-48bd-b65c-882694d24da6-istiod-ca-cert\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.979769 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.979491 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-podinfo\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.979769 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.979579 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-credential-socket\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.979769 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.979711 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-workload-certs\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.980020 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.979976 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-data\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.980020 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.980008 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-workload-socket\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.980098 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.980079 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-credential-socket\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.980214 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.980193 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8234dfb9-e2f4-48bd-b65c-882694d24da6-istiod-ca-cert\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.981788 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.981769 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-envoy\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.982015 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.981999 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-podinfo\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.992009 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.991973 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-token\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:33.992141 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:33.992119 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bptmz\" (UniqueName: \"kubernetes.io/projected/8234dfb9-e2f4-48bd-b65c-882694d24da6-kube-api-access-bptmz\") pod \"e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:34.138771 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:34.138705 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:34.299050 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:34.299020 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc"] Jul 06 23:50:34.300228 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:50:34.300202 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8234dfb9_e2f4_48bd_b65c_882694d24da6.slice/crio-130bc0235f09b21774432b6c7cf3d9db7f5eb7f9c1ef7eb038345a1154d3fc13 WatchSource:0}: Error finding container 130bc0235f09b21774432b6c7cf3d9db7f5eb7f9c1ef7eb038345a1154d3fc13: Status 404 returned error can't find the container with id 130bc0235f09b21774432b6c7cf3d9db7f5eb7f9c1ef7eb038345a1154d3fc13 Jul 06 23:50:34.308156 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:34.302650 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:50:34.308156 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:34.302841 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:50:34.308156 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:34.302936 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:50:34.531176 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:34.531134 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" event={"ID":"8234dfb9-e2f4-48bd-b65c-882694d24da6","Type":"ContainerStarted","Data":"c0e1fa5e54caefb1199684f8c8bd969a1dacec19e12cc0fbeb84228603eea78d"} Jul 06 23:50:34.531616 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:34.531242 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" event={"ID":"8234dfb9-e2f4-48bd-b65c-882694d24da6","Type":"ContainerStarted","Data":"130bc0235f09b21774432b6c7cf3d9db7f5eb7f9c1ef7eb038345a1154d3fc13"} Jul 06 23:50:34.532589 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:34.532573 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" Jul 06 23:50:34.532589 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:34.532579 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5" event={"ID":"3bdd578e-f2f4-4182-b0cc-0078aa98e2d1","Type":"ContainerDied","Data":"c12f320c158176ceb9bf3f9b58e5c10535c6dc4530fe291bf525a7a95e79e5b0"} Jul 06 23:50:34.532795 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:34.532630 2571 scope.go:117] "RemoveContainer" containerID="fed04a17b7bc529c7a60927f30fcd00e5611cd41249b5d82e840458cb54471f8" Jul 06 23:50:34.564550 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:34.564446 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" podStartSLOduration=1.5644301409999999 podStartE2EDuration="1.564430141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:34.561601714 +0000 UTC m=+1545.275656286" watchObservedRunningTime="2026-07-06 23:50:34.564430141 +0000 UTC m=+1545.278484711" Jul 06 23:50:34.580761 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:34.580707 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5"] Jul 06 23:50:34.584311 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:34.584287 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e6a06da5-openshift-default-74888cdf79-btzv5"] Jul 06 23:50:35.139229 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:35.139191 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:35.144643 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:35.144616 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:35.540142 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:35.540118 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:35.541235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:35.541214 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:35.786761 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:35.786699 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bdd578e-f2f4-4182-b0cc-0078aa98e2d1" path="/var/lib/kubelet/pods/3bdd578e-f2f4-4182-b0cc-0078aa98e2d1/volumes" Jul 06 23:50:39.860741 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:39.860685 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc"] Jul 06 23:50:39.861157 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:39.860884 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" podUID="8234dfb9-e2f4-48bd-b65c-882694d24da6" containerName="istio-proxy" containerID="cri-o://c0e1fa5e54caefb1199684f8c8bd969a1dacec19e12cc0fbeb84228603eea78d" gracePeriod=30 Jul 06 23:50:45.207273 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.207251 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:45.289569 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.289504 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-data\") pod \"8234dfb9-e2f4-48bd-b65c-882694d24da6\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " Jul 06 23:50:45.289569 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.289543 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bptmz\" (UniqueName: \"kubernetes.io/projected/8234dfb9-e2f4-48bd-b65c-882694d24da6-kube-api-access-bptmz\") pod \"8234dfb9-e2f4-48bd-b65c-882694d24da6\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " Jul 06 23:50:45.289569 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.289561 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-credential-socket\") pod \"8234dfb9-e2f4-48bd-b65c-882694d24da6\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " Jul 06 23:50:45.289855 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.289586 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-workload-certs\") pod \"8234dfb9-e2f4-48bd-b65c-882694d24da6\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " Jul 06 23:50:45.289855 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.289609 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-envoy\") pod \"8234dfb9-e2f4-48bd-b65c-882694d24da6\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " Jul 06 23:50:45.289855 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.289634 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-workload-socket\") pod \"8234dfb9-e2f4-48bd-b65c-882694d24da6\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " Jul 06 23:50:45.289855 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.289665 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-token\") pod \"8234dfb9-e2f4-48bd-b65c-882694d24da6\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " Jul 06 23:50:45.289855 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.289744 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-podinfo\") pod \"8234dfb9-e2f4-48bd-b65c-882694d24da6\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " Jul 06 23:50:45.289855 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.289777 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8234dfb9-e2f4-48bd-b65c-882694d24da6-istiod-ca-cert\") pod \"8234dfb9-e2f4-48bd-b65c-882694d24da6\" (UID: \"8234dfb9-e2f4-48bd-b65c-882694d24da6\") " Jul 06 23:50:45.290157 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.289848 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8234dfb9-e2f4-48bd-b65c-882694d24da6" (UID: "8234dfb9-e2f4-48bd-b65c-882694d24da6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:50:45.290157 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.289956 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8234dfb9-e2f4-48bd-b65c-882694d24da6" (UID: "8234dfb9-e2f4-48bd-b65c-882694d24da6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:50:45.290157 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.290085 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-credential-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:50:45.290157 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.290105 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-workload-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:50:45.290525 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.290209 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8234dfb9-e2f4-48bd-b65c-882694d24da6" (UID: "8234dfb9-e2f4-48bd-b65c-882694d24da6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:50:45.290525 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.290260 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8234dfb9-e2f4-48bd-b65c-882694d24da6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8234dfb9-e2f4-48bd-b65c-882694d24da6" (UID: "8234dfb9-e2f4-48bd-b65c-882694d24da6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:50:45.290525 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.290418 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-data" (OuterVolumeSpecName: "istio-data") pod "8234dfb9-e2f4-48bd-b65c-882694d24da6" (UID: "8234dfb9-e2f4-48bd-b65c-882694d24da6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:50:45.291942 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.291918 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8234dfb9-e2f4-48bd-b65c-882694d24da6-kube-api-access-bptmz" (OuterVolumeSpecName: "kube-api-access-bptmz") pod "8234dfb9-e2f4-48bd-b65c-882694d24da6" (UID: "8234dfb9-e2f4-48bd-b65c-882694d24da6"). InnerVolumeSpecName "kube-api-access-bptmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:50:45.292125 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.292106 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-token" (OuterVolumeSpecName: "istio-token") pod "8234dfb9-e2f4-48bd-b65c-882694d24da6" (UID: "8234dfb9-e2f4-48bd-b65c-882694d24da6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:50:45.292125 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.292115 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8234dfb9-e2f4-48bd-b65c-882694d24da6" (UID: "8234dfb9-e2f4-48bd-b65c-882694d24da6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 23:50:45.292219 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.292163 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8234dfb9-e2f4-48bd-b65c-882694d24da6" (UID: "8234dfb9-e2f4-48bd-b65c-882694d24da6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:50:45.391424 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.391401 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-data\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:50:45.391424 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.391423 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bptmz\" (UniqueName: \"kubernetes.io/projected/8234dfb9-e2f4-48bd-b65c-882694d24da6-kube-api-access-bptmz\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:50:45.391553 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.391432 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-envoy\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:50:45.391553 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.391441 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8234dfb9-e2f4-48bd-b65c-882694d24da6-workload-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:50:45.391553 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.391448 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-token\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:50:45.391553 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.391456 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8234dfb9-e2f4-48bd-b65c-882694d24da6-istio-podinfo\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:50:45.391553 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.391463 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8234dfb9-e2f4-48bd-b65c-882694d24da6-istiod-ca-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:50:45.592451 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.592348 2571 generic.go:356] "Generic (PLEG): container finished" podID="8234dfb9-e2f4-48bd-b65c-882694d24da6" containerID="c0e1fa5e54caefb1199684f8c8bd969a1dacec19e12cc0fbeb84228603eea78d" exitCode=0 Jul 06 23:50:45.592451 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.592409 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" Jul 06 23:50:45.592451 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.592442 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" event={"ID":"8234dfb9-e2f4-48bd-b65c-882694d24da6","Type":"ContainerDied","Data":"c0e1fa5e54caefb1199684f8c8bd969a1dacec19e12cc0fbeb84228603eea78d"} Jul 06 23:50:45.592706 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.592519 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc" event={"ID":"8234dfb9-e2f4-48bd-b65c-882694d24da6","Type":"ContainerDied","Data":"130bc0235f09b21774432b6c7cf3d9db7f5eb7f9c1ef7eb038345a1154d3fc13"} Jul 06 23:50:45.592706 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.592541 2571 scope.go:117] "RemoveContainer" containerID="c0e1fa5e54caefb1199684f8c8bd969a1dacec19e12cc0fbeb84228603eea78d" Jul 06 23:50:45.602651 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.602635 2571 scope.go:117] "RemoveContainer" containerID="c0e1fa5e54caefb1199684f8c8bd969a1dacec19e12cc0fbeb84228603eea78d" Jul 06 23:50:45.602917 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:50:45.602898 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e1fa5e54caefb1199684f8c8bd969a1dacec19e12cc0fbeb84228603eea78d\": container with ID starting with c0e1fa5e54caefb1199684f8c8bd969a1dacec19e12cc0fbeb84228603eea78d not found: ID does not exist" containerID="c0e1fa5e54caefb1199684f8c8bd969a1dacec19e12cc0fbeb84228603eea78d" Jul 06 23:50:45.602969 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.602924 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e1fa5e54caefb1199684f8c8bd969a1dacec19e12cc0fbeb84228603eea78d"} err="failed to get container status \"c0e1fa5e54caefb1199684f8c8bd969a1dacec19e12cc0fbeb84228603eea78d\": rpc error: code = NotFound desc = could not find container \"c0e1fa5e54caefb1199684f8c8bd969a1dacec19e12cc0fbeb84228603eea78d\": container with ID starting with c0e1fa5e54caefb1199684f8c8bd969a1dacec19e12cc0fbeb84228603eea78d not found: ID does not exist" Jul 06 23:50:45.621304 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.621278 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc"] Jul 06 23:50:45.640079 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.640055 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-5e8d5579-gw-openshift-default-5579899b6c-4pjlc"] Jul 06 23:50:45.786221 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:45.786190 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8234dfb9-e2f4-48bd-b65c-882694d24da6" path="/var/lib/kubelet/pods/8234dfb9-e2f4-48bd-b65c-882694d24da6/volumes" Jul 06 23:50:46.086779 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.086746 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8"] Jul 06 23:50:46.087275 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.087255 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8234dfb9-e2f4-48bd-b65c-882694d24da6" containerName="istio-proxy" Jul 06 23:50:46.087351 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.087279 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8234dfb9-e2f4-48bd-b65c-882694d24da6" containerName="istio-proxy" Jul 06 23:50:46.087409 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.087396 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8234dfb9-e2f4-48bd-b65c-882694d24da6" containerName="istio-proxy" Jul 06 23:50:46.091016 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.090990 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.093364 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.093346 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b1b9f5c3-openshift-default-dockercfg-2gnhk\"" Jul 06 23:50:46.102602 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.102582 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8"] Jul 06 23:50:46.198087 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.198059 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpmxp\" (UniqueName: \"kubernetes.io/projected/c94ee63e-1193-4118-a72d-123448a00baa-kube-api-access-zpmxp\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.198247 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.198091 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-workload-socket\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.198247 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.198147 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94ee63e-1193-4118-a72d-123448a00baa-istio-token\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.198247 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.198177 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94ee63e-1193-4118-a72d-123448a00baa-istiod-ca-cert\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.198247 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.198200 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-credential-socket\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.198247 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.198221 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-istio-envoy\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.198415 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.198287 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-workload-certs\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.198415 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.198323 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-istio-data\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.198415 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.198346 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94ee63e-1193-4118-a72d-123448a00baa-istio-podinfo\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.299157 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.299131 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpmxp\" (UniqueName: \"kubernetes.io/projected/c94ee63e-1193-4118-a72d-123448a00baa-kube-api-access-zpmxp\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.299508 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.299162 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-workload-socket\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.299508 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.299191 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94ee63e-1193-4118-a72d-123448a00baa-istio-token\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.299508 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.299379 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94ee63e-1193-4118-a72d-123448a00baa-istiod-ca-cert\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.299508 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.299419 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-credential-socket\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.299508 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.299444 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-istio-envoy\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.299508 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.299483 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-workload-certs\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.299508 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.299508 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-istio-data\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.299820 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.299529 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94ee63e-1193-4118-a72d-123448a00baa-istio-podinfo\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.299820 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.299538 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-workload-socket\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.299820 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.299771 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-credential-socket\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.299979 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.299873 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-istio-data\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.299979 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.299934 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-workload-certs\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.300088 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.300051 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94ee63e-1193-4118-a72d-123448a00baa-istiod-ca-cert\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.301742 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.301707 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-istio-envoy\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.302028 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.302011 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94ee63e-1193-4118-a72d-123448a00baa-istio-podinfo\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.307121 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.307094 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94ee63e-1193-4118-a72d-123448a00baa-istio-token\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.307210 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.307143 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpmxp\" (UniqueName: \"kubernetes.io/projected/c94ee63e-1193-4118-a72d-123448a00baa-kube-api-access-zpmxp\") pod \"e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.403397 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.403346 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:46.536324 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.536296 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8"] Jul 06 23:50:46.539441 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:50:46.539410 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94ee63e_1193_4118_a72d_123448a00baa.slice/crio-2d82822bc5033590b11ce813573f9bb407ae1d0e5710178a8103f710134587cf WatchSource:0}: Error finding container 2d82822bc5033590b11ce813573f9bb407ae1d0e5710178a8103f710134587cf: Status 404 returned error can't find the container with id 2d82822bc5033590b11ce813573f9bb407ae1d0e5710178a8103f710134587cf Jul 06 23:50:46.542447 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.542416 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:50:46.542517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.542481 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:50:46.542517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.542508 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:50:46.601152 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:46.601128 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" event={"ID":"c94ee63e-1193-4118-a72d-123448a00baa","Type":"ContainerStarted","Data":"2d82822bc5033590b11ce813573f9bb407ae1d0e5710178a8103f710134587cf"} Jul 06 23:50:47.607555 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:47.607521 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" event={"ID":"c94ee63e-1193-4118-a72d-123448a00baa","Type":"ContainerStarted","Data":"966f983e774e868d9319e2d959e1134408229259986056ecfd4d2a567ff8591f"} Jul 06 23:50:47.628364 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:47.628296 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" podStartSLOduration=1.628277091 podStartE2EDuration="1.628277091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:50:47.627063644 +0000 UTC m=+1558.341118214" watchObservedRunningTime="2026-07-06 23:50:47.628277091 +0000 UTC m=+1558.342331664" Jul 06 23:50:48.403788 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:48.403751 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:48.408510 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:48.408483 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:48.612985 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:48.612946 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:48.613972 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:48.613944 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:57.671286 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:57.671252 2571 generic.go:356] "Generic (PLEG): container finished" podID="c94ee63e-1193-4118-a72d-123448a00baa" containerID="966f983e774e868d9319e2d959e1134408229259986056ecfd4d2a567ff8591f" exitCode=0 Jul 06 23:50:57.671683 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:57.671284 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" event={"ID":"c94ee63e-1193-4118-a72d-123448a00baa","Type":"ContainerDied","Data":"966f983e774e868d9319e2d959e1134408229259986056ecfd4d2a567ff8591f"} Jul 06 23:50:57.671683 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:57.671680 2571 scope.go:117] "RemoveContainer" containerID="966f983e774e868d9319e2d959e1134408229259986056ecfd4d2a567ff8591f" Jul 06 23:50:57.672696 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:57.672661 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:50:57.672785 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:57.672773 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:50:57.672832 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:57.672811 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:50:58.682207 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:58.682172 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" event={"ID":"c94ee63e-1193-4118-a72d-123448a00baa","Type":"ContainerStarted","Data":"055f5232ed65181bcf778513d3b77841257ecb00ab34aa1043245c7df5d47838"} Jul 06 23:50:58.776487 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:58.776454 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8"] Jul 06 23:50:59.404332 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:59.404294 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:59.409268 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:59.409237 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:59.687508 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:59.687474 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" podUID="c94ee63e-1193-4118-a72d-123448a00baa" containerName="istio-proxy" containerID="cri-o://055f5232ed65181bcf778513d3b77841257ecb00ab34aa1043245c7df5d47838" gracePeriod=30 Jul 06 23:50:59.687958 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:59.687596 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:50:59.688671 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:50:59.688647 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:51:04.938802 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:04.938778 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:51:05.001868 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.001760 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q"] Jul 06 23:51:05.003129 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.003105 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c94ee63e-1193-4118-a72d-123448a00baa" containerName="istio-proxy" Jul 06 23:51:05.003290 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.003276 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c94ee63e-1193-4118-a72d-123448a00baa" containerName="istio-proxy" Jul 06 23:51:05.003403 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.003393 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c94ee63e-1193-4118-a72d-123448a00baa" containerName="istio-proxy" Jul 06 23:51:05.003483 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.003474 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c94ee63e-1193-4118-a72d-123448a00baa" containerName="istio-proxy" Jul 06 23:51:05.003817 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.003802 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c94ee63e-1193-4118-a72d-123448a00baa" containerName="istio-proxy" Jul 06 23:51:05.003934 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.003924 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c94ee63e-1193-4118-a72d-123448a00baa" containerName="istio-proxy" Jul 06 23:51:05.006993 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.006970 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.009998 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.009973 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-325feefe-openshift-default-dockercfg-t2vw9\"" Jul 06 23:51:05.017037 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.017018 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q"] Jul 06 23:51:05.071140 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.071112 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94ee63e-1193-4118-a72d-123448a00baa-istio-token\") pod \"c94ee63e-1193-4118-a72d-123448a00baa\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " Jul 06 23:51:05.071286 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.071169 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94ee63e-1193-4118-a72d-123448a00baa-istiod-ca-cert\") pod \"c94ee63e-1193-4118-a72d-123448a00baa\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " Jul 06 23:51:05.071286 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.071214 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-workload-socket\") pod \"c94ee63e-1193-4118-a72d-123448a00baa\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " Jul 06 23:51:05.071286 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.071237 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-istio-envoy\") pod \"c94ee63e-1193-4118-a72d-123448a00baa\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " Jul 06 23:51:05.071286 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.071269 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-istio-data\") pod \"c94ee63e-1193-4118-a72d-123448a00baa\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " Jul 06 23:51:05.071493 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.071318 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-workload-certs\") pod \"c94ee63e-1193-4118-a72d-123448a00baa\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " Jul 06 23:51:05.071493 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.071387 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpmxp\" (UniqueName: \"kubernetes.io/projected/c94ee63e-1193-4118-a72d-123448a00baa-kube-api-access-zpmxp\") pod \"c94ee63e-1193-4118-a72d-123448a00baa\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " Jul 06 23:51:05.071493 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.071419 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-credential-socket\") pod \"c94ee63e-1193-4118-a72d-123448a00baa\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " Jul 06 23:51:05.071493 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.071456 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94ee63e-1193-4118-a72d-123448a00baa-istio-podinfo\") pod \"c94ee63e-1193-4118-a72d-123448a00baa\" (UID: \"c94ee63e-1193-4118-a72d-123448a00baa\") " Jul 06 23:51:05.071692 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.071584 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c94ee63e-1193-4118-a72d-123448a00baa" (UID: "c94ee63e-1193-4118-a72d-123448a00baa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:51:05.071775 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.071702 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94ee63e-1193-4118-a72d-123448a00baa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c94ee63e-1193-4118-a72d-123448a00baa" (UID: "c94ee63e-1193-4118-a72d-123448a00baa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:51:05.071840 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.071795 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-workload-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:51:05.071840 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.071816 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94ee63e-1193-4118-a72d-123448a00baa-istiod-ca-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:51:05.071840 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.071824 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c94ee63e-1193-4118-a72d-123448a00baa" (UID: "c94ee63e-1193-4118-a72d-123448a00baa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:51:05.071989 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.071914 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c94ee63e-1193-4118-a72d-123448a00baa" (UID: "c94ee63e-1193-4118-a72d-123448a00baa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:51:05.072569 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.072452 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-istio-data" (OuterVolumeSpecName: "istio-data") pod "c94ee63e-1193-4118-a72d-123448a00baa" (UID: "c94ee63e-1193-4118-a72d-123448a00baa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:51:05.073610 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.073579 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c94ee63e-1193-4118-a72d-123448a00baa" (UID: "c94ee63e-1193-4118-a72d-123448a00baa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:51:05.073756 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.073735 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94ee63e-1193-4118-a72d-123448a00baa-kube-api-access-zpmxp" (OuterVolumeSpecName: "kube-api-access-zpmxp") pod "c94ee63e-1193-4118-a72d-123448a00baa" (UID: "c94ee63e-1193-4118-a72d-123448a00baa"). InnerVolumeSpecName "kube-api-access-zpmxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:51:05.073888 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.073871 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94ee63e-1193-4118-a72d-123448a00baa-istio-token" (OuterVolumeSpecName: "istio-token") pod "c94ee63e-1193-4118-a72d-123448a00baa" (UID: "c94ee63e-1193-4118-a72d-123448a00baa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:51:05.074006 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.073992 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c94ee63e-1193-4118-a72d-123448a00baa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c94ee63e-1193-4118-a72d-123448a00baa" (UID: "c94ee63e-1193-4118-a72d-123448a00baa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 23:51:05.172442 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.172402 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-credential-socket\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.172603 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.172447 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-data\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.172603 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.172543 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-podinfo\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.172603 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.172594 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-workload-certs\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.172830 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.172664 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-269vt\" (UniqueName: \"kubernetes.io/projected/4ea33c7a-9b1e-47a2-9e43-27efae212604-kube-api-access-269vt\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.172830 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.172691 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-envoy\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.172830 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.172764 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-token\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.172830 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.172814 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-workload-socket\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.172984 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.172866 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea33c7a-9b1e-47a2-9e43-27efae212604-istiod-ca-cert\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.172984 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.172909 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-istio-envoy\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:51:05.172984 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.172919 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-istio-data\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:51:05.172984 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.172928 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-workload-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:51:05.172984 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.172943 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpmxp\" (UniqueName: \"kubernetes.io/projected/c94ee63e-1193-4118-a72d-123448a00baa-kube-api-access-zpmxp\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:51:05.172984 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.172955 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94ee63e-1193-4118-a72d-123448a00baa-credential-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:51:05.172984 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.172964 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94ee63e-1193-4118-a72d-123448a00baa-istio-podinfo\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:51:05.172984 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.172973 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94ee63e-1193-4118-a72d-123448a00baa-istio-token\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:51:05.273992 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.273920 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-workload-socket\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.273992 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.273954 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea33c7a-9b1e-47a2-9e43-27efae212604-istiod-ca-cert\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.273992 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.273986 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-credential-socket\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.274263 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.274111 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-data\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.274263 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.274178 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-podinfo\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.274263 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.274220 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-workload-certs\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.274412 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.274281 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-269vt\" (UniqueName: \"kubernetes.io/projected/4ea33c7a-9b1e-47a2-9e43-27efae212604-kube-api-access-269vt\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.274412 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.274314 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-envoy\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.274412 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.274340 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-credential-socket\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.274412 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.274350 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-token\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.274596 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.274285 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-workload-socket\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.274596 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.274463 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-data\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.274596 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.274471 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-workload-certs\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.274816 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.274741 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea33c7a-9b1e-47a2-9e43-27efae212604-istiod-ca-cert\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.276648 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.276622 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-envoy\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.276810 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.276791 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-podinfo\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.281509 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.281483 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-token\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.281651 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.281635 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-269vt\" (UniqueName: \"kubernetes.io/projected/4ea33c7a-9b1e-47a2-9e43-27efae212604-kube-api-access-269vt\") pod \"e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.319525 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.319507 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:05.480746 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.480696 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q"] Jul 06 23:51:05.482267 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:51:05.482240 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea33c7a_9b1e_47a2_9e43_27efae212604.slice/crio-673e4fadc89b36c8962339c676a769cf51f62cbc8f7bcbd2c0e2650b12e86f00 WatchSource:0}: Error finding container 673e4fadc89b36c8962339c676a769cf51f62cbc8f7bcbd2c0e2650b12e86f00: Status 404 returned error can't find the container with id 673e4fadc89b36c8962339c676a769cf51f62cbc8f7bcbd2c0e2650b12e86f00 Jul 06 23:51:05.484322 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.484290 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:51:05.484403 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.484353 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:51:05.484403 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.484384 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:51:05.722949 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.722905 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" event={"ID":"4ea33c7a-9b1e-47a2-9e43-27efae212604","Type":"ContainerStarted","Data":"845cb843f9c02ba141841986fb6dd26e308776d6d6f07d96e49e35f05645fe46"} Jul 06 23:51:05.723125 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.722995 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" event={"ID":"4ea33c7a-9b1e-47a2-9e43-27efae212604","Type":"ContainerStarted","Data":"673e4fadc89b36c8962339c676a769cf51f62cbc8f7bcbd2c0e2650b12e86f00"} Jul 06 23:51:05.724696 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.724668 2571 generic.go:356] "Generic (PLEG): container finished" podID="c94ee63e-1193-4118-a72d-123448a00baa" containerID="055f5232ed65181bcf778513d3b77841257ecb00ab34aa1043245c7df5d47838" exitCode=0 Jul 06 23:51:05.724850 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.724712 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" event={"ID":"c94ee63e-1193-4118-a72d-123448a00baa","Type":"ContainerDied","Data":"055f5232ed65181bcf778513d3b77841257ecb00ab34aa1043245c7df5d47838"} Jul 06 23:51:05.724850 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.724752 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" Jul 06 23:51:05.724850 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.724769 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8" event={"ID":"c94ee63e-1193-4118-a72d-123448a00baa","Type":"ContainerDied","Data":"2d82822bc5033590b11ce813573f9bb407ae1d0e5710178a8103f710134587cf"} Jul 06 23:51:05.724850 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.724791 2571 scope.go:117] "RemoveContainer" containerID="055f5232ed65181bcf778513d3b77841257ecb00ab34aa1043245c7df5d47838" Jul 06 23:51:05.734844 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.734826 2571 scope.go:117] "RemoveContainer" containerID="966f983e774e868d9319e2d959e1134408229259986056ecfd4d2a567ff8591f" Jul 06 23:51:05.743315 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.743299 2571 scope.go:117] "RemoveContainer" containerID="055f5232ed65181bcf778513d3b77841257ecb00ab34aa1043245c7df5d47838" Jul 06 23:51:05.743633 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:51:05.743607 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055f5232ed65181bcf778513d3b77841257ecb00ab34aa1043245c7df5d47838\": container with ID starting with 055f5232ed65181bcf778513d3b77841257ecb00ab34aa1043245c7df5d47838 not found: ID does not exist" containerID="055f5232ed65181bcf778513d3b77841257ecb00ab34aa1043245c7df5d47838" Jul 06 23:51:05.743765 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.743644 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055f5232ed65181bcf778513d3b77841257ecb00ab34aa1043245c7df5d47838"} err="failed to get container status \"055f5232ed65181bcf778513d3b77841257ecb00ab34aa1043245c7df5d47838\": rpc error: code = NotFound desc = could not find container \"055f5232ed65181bcf778513d3b77841257ecb00ab34aa1043245c7df5d47838\": container with ID starting with 055f5232ed65181bcf778513d3b77841257ecb00ab34aa1043245c7df5d47838 not found: ID does not exist" Jul 06 23:51:05.743765 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.743666 2571 scope.go:117] "RemoveContainer" containerID="966f983e774e868d9319e2d959e1134408229259986056ecfd4d2a567ff8591f" Jul 06 23:51:05.744037 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:51:05.743979 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966f983e774e868d9319e2d959e1134408229259986056ecfd4d2a567ff8591f\": container with ID starting with 966f983e774e868d9319e2d959e1134408229259986056ecfd4d2a567ff8591f not found: ID does not exist" containerID="966f983e774e868d9319e2d959e1134408229259986056ecfd4d2a567ff8591f" Jul 06 23:51:05.744037 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.744009 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966f983e774e868d9319e2d959e1134408229259986056ecfd4d2a567ff8591f"} err="failed to get container status \"966f983e774e868d9319e2d959e1134408229259986056ecfd4d2a567ff8591f\": rpc error: code = NotFound desc = could not find container \"966f983e774e868d9319e2d959e1134408229259986056ecfd4d2a567ff8591f\": container with ID starting with 966f983e774e868d9319e2d959e1134408229259986056ecfd4d2a567ff8591f not found: ID does not exist" Jul 06 23:51:05.746199 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.746152 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" podStartSLOduration=1.746141795 podStartE2EDuration="1.746141795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:51:05.744218077 +0000 UTC m=+1576.458272651" watchObservedRunningTime="2026-07-06 23:51:05.746141795 +0000 UTC m=+1576.460196365" Jul 06 23:51:05.758996 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.758961 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8"] Jul 06 23:51:05.761113 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.761075 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b1b9f5c3-openshift-default-5699757d49-qbjr8"] Jul 06 23:51:05.785559 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:05.785508 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c94ee63e-1193-4118-a72d-123448a00baa" path="/var/lib/kubelet/pods/c94ee63e-1193-4118-a72d-123448a00baa/volumes" Jul 06 23:51:06.320637 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:06.320607 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:06.335130 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:06.335100 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:06.731517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:06.731492 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:06.732598 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:06.732579 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:10.812221 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:10.812181 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q"] Jul 06 23:51:10.812785 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:10.812422 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" podUID="4ea33c7a-9b1e-47a2-9e43-27efae212604" containerName="istio-proxy" containerID="cri-o://845cb843f9c02ba141841986fb6dd26e308776d6d6f07d96e49e35f05645fe46" gracePeriod=30 Jul 06 23:51:17.366479 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.366458 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:17.388899 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.388844 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-credential-socket\") pod \"4ea33c7a-9b1e-47a2-9e43-27efae212604\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " Jul 06 23:51:17.388899 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.388887 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea33c7a-9b1e-47a2-9e43-27efae212604-istiod-ca-cert\") pod \"4ea33c7a-9b1e-47a2-9e43-27efae212604\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " Jul 06 23:51:17.389052 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.388907 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-token\") pod \"4ea33c7a-9b1e-47a2-9e43-27efae212604\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " Jul 06 23:51:17.389052 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.388942 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-podinfo\") pod \"4ea33c7a-9b1e-47a2-9e43-27efae212604\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " Jul 06 23:51:17.389052 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.388974 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-data\") pod \"4ea33c7a-9b1e-47a2-9e43-27efae212604\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " Jul 06 23:51:17.389052 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.388998 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-workload-certs\") pod \"4ea33c7a-9b1e-47a2-9e43-27efae212604\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " Jul 06 23:51:17.389052 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.389034 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-269vt\" (UniqueName: \"kubernetes.io/projected/4ea33c7a-9b1e-47a2-9e43-27efae212604-kube-api-access-269vt\") pod \"4ea33c7a-9b1e-47a2-9e43-27efae212604\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " Jul 06 23:51:17.389307 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.389070 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-envoy\") pod \"4ea33c7a-9b1e-47a2-9e43-27efae212604\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " Jul 06 23:51:17.389307 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.389097 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-workload-socket\") pod \"4ea33c7a-9b1e-47a2-9e43-27efae212604\" (UID: \"4ea33c7a-9b1e-47a2-9e43-27efae212604\") " Jul 06 23:51:17.389307 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.389192 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea33c7a-9b1e-47a2-9e43-27efae212604-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4ea33c7a-9b1e-47a2-9e43-27efae212604" (UID: "4ea33c7a-9b1e-47a2-9e43-27efae212604"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:51:17.389307 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.389233 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4ea33c7a-9b1e-47a2-9e43-27efae212604" (UID: "4ea33c7a-9b1e-47a2-9e43-27efae212604"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:51:17.389504 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.389396 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4ea33c7a-9b1e-47a2-9e43-27efae212604" (UID: "4ea33c7a-9b1e-47a2-9e43-27efae212604"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:51:17.389504 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.389412 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4ea33c7a-9b1e-47a2-9e43-27efae212604" (UID: "4ea33c7a-9b1e-47a2-9e43-27efae212604"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:51:17.389616 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.389503 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-credential-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:51:17.389616 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.389523 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea33c7a-9b1e-47a2-9e43-27efae212604-istiod-ca-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:51:17.391639 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.391601 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-data" (OuterVolumeSpecName: "istio-data") pod "4ea33c7a-9b1e-47a2-9e43-27efae212604" (UID: "4ea33c7a-9b1e-47a2-9e43-27efae212604"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:51:17.391936 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.391916 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-token" (OuterVolumeSpecName: "istio-token") pod "4ea33c7a-9b1e-47a2-9e43-27efae212604" (UID: "4ea33c7a-9b1e-47a2-9e43-27efae212604"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:51:17.392151 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.392134 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea33c7a-9b1e-47a2-9e43-27efae212604-kube-api-access-269vt" (OuterVolumeSpecName: "kube-api-access-269vt") pod "4ea33c7a-9b1e-47a2-9e43-27efae212604" (UID: "4ea33c7a-9b1e-47a2-9e43-27efae212604"). InnerVolumeSpecName "kube-api-access-269vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:51:17.392235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.392148 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4ea33c7a-9b1e-47a2-9e43-27efae212604" (UID: "4ea33c7a-9b1e-47a2-9e43-27efae212604"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 23:51:17.392658 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.392622 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4ea33c7a-9b1e-47a2-9e43-27efae212604" (UID: "4ea33c7a-9b1e-47a2-9e43-27efae212604"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:51:17.490907 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.490868 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-token\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:51:17.490907 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.490905 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-podinfo\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:51:17.490907 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.490914 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-data\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:51:17.490907 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.490922 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-workload-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:51:17.491125 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.490929 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-269vt\" (UniqueName: \"kubernetes.io/projected/4ea33c7a-9b1e-47a2-9e43-27efae212604-kube-api-access-269vt\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:51:17.491125 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.490940 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-istio-envoy\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:51:17.491125 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.490948 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea33c7a-9b1e-47a2-9e43-27efae212604-workload-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:51:17.790265 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.790229 2571 generic.go:356] "Generic (PLEG): container finished" podID="4ea33c7a-9b1e-47a2-9e43-27efae212604" containerID="845cb843f9c02ba141841986fb6dd26e308776d6d6f07d96e49e35f05645fe46" exitCode=0 Jul 06 23:51:17.790265 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.790265 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" Jul 06 23:51:17.790467 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.790303 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" event={"ID":"4ea33c7a-9b1e-47a2-9e43-27efae212604","Type":"ContainerDied","Data":"845cb843f9c02ba141841986fb6dd26e308776d6d6f07d96e49e35f05645fe46"} Jul 06 23:51:17.790467 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.790351 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q" event={"ID":"4ea33c7a-9b1e-47a2-9e43-27efae212604","Type":"ContainerDied","Data":"673e4fadc89b36c8962339c676a769cf51f62cbc8f7bcbd2c0e2650b12e86f00"} Jul 06 23:51:17.790467 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.790374 2571 scope.go:117] "RemoveContainer" containerID="845cb843f9c02ba141841986fb6dd26e308776d6d6f07d96e49e35f05645fe46" Jul 06 23:51:17.802388 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.802372 2571 scope.go:117] "RemoveContainer" containerID="845cb843f9c02ba141841986fb6dd26e308776d6d6f07d96e49e35f05645fe46" Jul 06 23:51:17.802780 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:51:17.802695 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845cb843f9c02ba141841986fb6dd26e308776d6d6f07d96e49e35f05645fe46\": container with ID starting with 845cb843f9c02ba141841986fb6dd26e308776d6d6f07d96e49e35f05645fe46 not found: ID does not exist" containerID="845cb843f9c02ba141841986fb6dd26e308776d6d6f07d96e49e35f05645fe46" Jul 06 23:51:17.802780 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.802748 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845cb843f9c02ba141841986fb6dd26e308776d6d6f07d96e49e35f05645fe46"} err="failed to get container status \"845cb843f9c02ba141841986fb6dd26e308776d6d6f07d96e49e35f05645fe46\": rpc error: code = NotFound desc = could not find container \"845cb843f9c02ba141841986fb6dd26e308776d6d6f07d96e49e35f05645fe46\": container with ID starting with 845cb843f9c02ba141841986fb6dd26e308776d6d6f07d96e49e35f05645fe46 not found: ID does not exist" Jul 06 23:51:17.815394 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.815371 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q"] Jul 06 23:51:17.823594 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:17.823574 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-325feefe-openshift-default-64b6bfc575-r8h4q"] Jul 06 23:51:19.786445 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:51:19.786416 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ea33c7a-9b1e-47a2-9e43-27efae212604" path="/var/lib/kubelet/pods/4ea33c7a-9b1e-47a2-9e43-27efae212604/volumes" Jul 06 23:52:52.295907 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.295873 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck"] Jul 06 23:52:52.296428 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.296204 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea33c7a-9b1e-47a2-9e43-27efae212604" containerName="istio-proxy" Jul 06 23:52:52.296428 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.296215 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea33c7a-9b1e-47a2-9e43-27efae212604" containerName="istio-proxy" Jul 06 23:52:52.296428 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.296290 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea33c7a-9b1e-47a2-9e43-27efae212604" containerName="istio-proxy" Jul 06 23:52:52.298177 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.298151 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.301392 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.301370 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-793bb430-openshift-default-dockercfg-x66cj\"" Jul 06 23:52:52.306208 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.306186 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-podinfo\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.306307 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.306223 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46n6j\" (UniqueName: \"kubernetes.io/projected/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-kube-api-access-46n6j\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.306307 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.306275 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-credential-socket\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.306396 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.306304 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-token\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.306396 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.306362 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istiod-ca-cert\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.306485 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.306402 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-workload-certs\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.306535 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.306491 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-workload-socket\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.306580 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.306534 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-data\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.306629 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.306609 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-envoy\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.327078 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.327048 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck"] Jul 06 23:52:52.407378 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.407346 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-token\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.407378 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.407387 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istiod-ca-cert\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.407651 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.407427 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-workload-certs\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.407651 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.407468 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-workload-socket\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.407651 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.407501 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-data\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.407651 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.407559 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-envoy\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.407651 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.407595 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-podinfo\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.407651 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.407620 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46n6j\" (UniqueName: \"kubernetes.io/projected/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-kube-api-access-46n6j\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.408037 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.407653 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-credential-socket\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.408101 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.408087 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-credential-socket\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.408293 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.408249 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-workload-certs\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.408509 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.408487 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-data\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.408616 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.408596 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-workload-socket\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.408812 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.408788 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istiod-ca-cert\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.410670 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.410640 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-envoy\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.411007 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.410988 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-podinfo\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.426019 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.425989 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46n6j\" (UniqueName: \"kubernetes.io/projected/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-kube-api-access-46n6j\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.426121 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.426032 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-token\") pod \"e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.608756 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.608644 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:52.754760 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.754709 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck"] Jul 06 23:52:52.757869 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:52:52.757838 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6f8ba3_8c39_4614_ba25_d377223fb5cb.slice/crio-7b9372e1787d9a20929483490f49610d1b36f6bb0cfb4c6f315edab9ba781375 WatchSource:0}: Error finding container 7b9372e1787d9a20929483490f49610d1b36f6bb0cfb4c6f315edab9ba781375: Status 404 returned error can't find the container with id 7b9372e1787d9a20929483490f49610d1b36f6bb0cfb4c6f315edab9ba781375 Jul 06 23:52:52.759954 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.759923 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:52:52.760038 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.759987 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:52:52.760038 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:52.760031 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:52:53.272902 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:53.272864 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" event={"ID":"2b6f8ba3-8c39-4614-ba25-d377223fb5cb","Type":"ContainerStarted","Data":"a1d2fd9751de08e095ee279b9a414fec9d9f331318e011be44266a6057b48856"} Jul 06 23:52:53.273081 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:53.272929 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" event={"ID":"2b6f8ba3-8c39-4614-ba25-d377223fb5cb","Type":"ContainerStarted","Data":"7b9372e1787d9a20929483490f49610d1b36f6bb0cfb4c6f315edab9ba781375"} Jul 06 23:52:53.293278 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:53.293231 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" podStartSLOduration=1.293217129 podStartE2EDuration="1.293217129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:52:53.291913187 +0000 UTC m=+1684.005967759" watchObservedRunningTime="2026-07-06 23:52:53.293217129 +0000 UTC m=+1684.007271769" Jul 06 23:52:53.609591 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:53.609500 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:53.614335 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:53.614312 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:54.278241 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:54.278207 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:52:54.279044 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:52:54.279025 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:53:09.279566 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:09.279531 2571 patch_prober.go:28] interesting pod/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 06 23:53:09.280019 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:09.279590 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" podUID="2b6f8ba3-8c39-4614-ba25-d377223fb5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 06 23:53:09.365988 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:09.365961 2571 generic.go:356] "Generic (PLEG): container finished" podID="2b6f8ba3-8c39-4614-ba25-d377223fb5cb" containerID="a1d2fd9751de08e095ee279b9a414fec9d9f331318e011be44266a6057b48856" exitCode=0 Jul 06 23:53:09.366150 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:09.366010 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" event={"ID":"2b6f8ba3-8c39-4614-ba25-d377223fb5cb","Type":"ContainerDied","Data":"a1d2fd9751de08e095ee279b9a414fec9d9f331318e011be44266a6057b48856"} Jul 06 23:53:09.366410 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:09.366394 2571 scope.go:117] "RemoveContainer" containerID="a1d2fd9751de08e095ee279b9a414fec9d9f331318e011be44266a6057b48856" Jul 06 23:53:09.367437 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:09.367403 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:53:09.367530 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:09.367477 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:53:09.367530 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:09.367506 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:53:10.372601 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:10.372516 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" event={"ID":"2b6f8ba3-8c39-4614-ba25-d377223fb5cb","Type":"ContainerStarted","Data":"25b4d06adc7b613ddb065a5f421dbebd4086e3f2a670561c42504243615bfe82"} Jul 06 23:53:10.609288 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:10.609248 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:53:10.609648 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:10.609616 2571 patch_prober.go:28] interesting pod/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 06 23:53:10.609776 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:10.609696 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" podUID="2b6f8ba3-8c39-4614-ba25-d377223fb5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 06 23:53:11.620743 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:11.619804 2571 patch_prober.go:28] interesting pod/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 06 23:53:11.620743 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:11.619863 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" podUID="2b6f8ba3-8c39-4614-ba25-d377223fb5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 06 23:53:12.609287 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:12.609248 2571 patch_prober.go:28] interesting pod/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 06 23:53:12.609588 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:12.609553 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" podUID="2b6f8ba3-8c39-4614-ba25-d377223fb5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 06 23:53:13.609138 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:13.609101 2571 patch_prober.go:28] interesting pod/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 06 23:53:13.609739 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:13.609678 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" podUID="2b6f8ba3-8c39-4614-ba25-d377223fb5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 06 23:53:13.878628 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:13.878544 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck"] Jul 06 23:53:13.878834 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:13.878808 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" podUID="2b6f8ba3-8c39-4614-ba25-d377223fb5cb" containerName="istio-proxy" containerID="cri-o://25b4d06adc7b613ddb065a5f421dbebd4086e3f2a670561c42504243615bfe82" gracePeriod=30 Jul 06 23:53:19.420921 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.420885 2571 generic.go:356] "Generic (PLEG): container finished" podID="2b6f8ba3-8c39-4614-ba25-d377223fb5cb" containerID="25b4d06adc7b613ddb065a5f421dbebd4086e3f2a670561c42504243615bfe82" exitCode=0 Jul 06 23:53:19.421366 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.420953 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" event={"ID":"2b6f8ba3-8c39-4614-ba25-d377223fb5cb","Type":"ContainerDied","Data":"25b4d06adc7b613ddb065a5f421dbebd4086e3f2a670561c42504243615bfe82"} Jul 06 23:53:19.421366 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.421013 2571 scope.go:117] "RemoveContainer" containerID="a1d2fd9751de08e095ee279b9a414fec9d9f331318e011be44266a6057b48856" Jul 06 23:53:19.540240 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.540216 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:53:19.639442 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.639356 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istiod-ca-cert\") pod \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " Jul 06 23:53:19.639442 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.639390 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-data\") pod \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " Jul 06 23:53:19.639442 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.639407 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-envoy\") pod \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " Jul 06 23:53:19.639748 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.639472 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-credential-socket\") pod \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " Jul 06 23:53:19.639748 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.639505 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-token\") pod \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " Jul 06 23:53:19.639748 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.639546 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-podinfo\") pod \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " Jul 06 23:53:19.639748 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.639566 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-workload-socket\") pod \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " Jul 06 23:53:19.639748 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.639621 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46n6j\" (UniqueName: \"kubernetes.io/projected/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-kube-api-access-46n6j\") pod \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " Jul 06 23:53:19.639748 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.639654 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-workload-certs\") pod \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\" (UID: \"2b6f8ba3-8c39-4614-ba25-d377223fb5cb\") " Jul 06 23:53:19.639748 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.639705 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2b6f8ba3-8c39-4614-ba25-d377223fb5cb" (UID: "2b6f8ba3-8c39-4614-ba25-d377223fb5cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:53:19.640120 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.639799 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2b6f8ba3-8c39-4614-ba25-d377223fb5cb" (UID: "2b6f8ba3-8c39-4614-ba25-d377223fb5cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:53:19.640120 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.640035 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2b6f8ba3-8c39-4614-ba25-d377223fb5cb" (UID: "2b6f8ba3-8c39-4614-ba25-d377223fb5cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:53:19.640120 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.640091 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2b6f8ba3-8c39-4614-ba25-d377223fb5cb" (UID: "2b6f8ba3-8c39-4614-ba25-d377223fb5cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:53:19.640280 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.640156 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-workload-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:53:19.640280 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.640175 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istiod-ca-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:53:19.640280 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.640189 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-credential-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:53:19.640280 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.640227 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-data" (OuterVolumeSpecName: "istio-data") pod "2b6f8ba3-8c39-4614-ba25-d377223fb5cb" (UID: "2b6f8ba3-8c39-4614-ba25-d377223fb5cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:53:19.642057 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.642023 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-kube-api-access-46n6j" (OuterVolumeSpecName: "kube-api-access-46n6j") pod "2b6f8ba3-8c39-4614-ba25-d377223fb5cb" (UID: "2b6f8ba3-8c39-4614-ba25-d377223fb5cb"). InnerVolumeSpecName "kube-api-access-46n6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:53:19.642170 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.642078 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2b6f8ba3-8c39-4614-ba25-d377223fb5cb" (UID: "2b6f8ba3-8c39-4614-ba25-d377223fb5cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:53:19.642545 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.642525 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2b6f8ba3-8c39-4614-ba25-d377223fb5cb" (UID: "2b6f8ba3-8c39-4614-ba25-d377223fb5cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 23:53:19.642617 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.642538 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-token" (OuterVolumeSpecName: "istio-token") pod "2b6f8ba3-8c39-4614-ba25-d377223fb5cb" (UID: "2b6f8ba3-8c39-4614-ba25-d377223fb5cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:53:19.741043 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.741016 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46n6j\" (UniqueName: \"kubernetes.io/projected/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-kube-api-access-46n6j\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:53:19.741043 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.741042 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-data\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:53:19.741192 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.741050 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-envoy\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:53:19.741192 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.741058 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-token\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:53:19.741192 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.741066 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-istio-podinfo\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:53:19.741192 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:19.741074 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6f8ba3-8c39-4614-ba25-d377223fb5cb-workload-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:53:20.428183 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:20.428158 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" Jul 06 23:53:20.428614 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:20.428182 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck" event={"ID":"2b6f8ba3-8c39-4614-ba25-d377223fb5cb","Type":"ContainerDied","Data":"7b9372e1787d9a20929483490f49610d1b36f6bb0cfb4c6f315edab9ba781375"} Jul 06 23:53:20.428614 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:20.428234 2571 scope.go:117] "RemoveContainer" containerID="25b4d06adc7b613ddb065a5f421dbebd4086e3f2a670561c42504243615bfe82" Jul 06 23:53:20.446900 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:20.446879 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck"] Jul 06 23:53:20.451339 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:20.451320 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-793bb430-openshift-default-7bdcd9d5f5-kdrck"] Jul 06 23:53:21.785959 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:21.785925 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b6f8ba3-8c39-4614-ba25-d377223fb5cb" path="/var/lib/kubelet/pods/2b6f8ba3-8c39-4614-ba25-d377223fb5cb/volumes" Jul 06 23:53:33.393331 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.393281 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg"] Jul 06 23:53:33.394080 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.394058 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6f8ba3-8c39-4614-ba25-d377223fb5cb" containerName="istio-proxy" Jul 06 23:53:33.394174 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.394084 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6f8ba3-8c39-4614-ba25-d377223fb5cb" containerName="istio-proxy" Jul 06 23:53:33.394234 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.394219 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6f8ba3-8c39-4614-ba25-d377223fb5cb" containerName="istio-proxy" Jul 06 23:53:33.394234 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.394230 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6f8ba3-8c39-4614-ba25-d377223fb5cb" containerName="istio-proxy" Jul 06 23:53:33.403977 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.403948 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.407058 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.407035 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5443d911-openshift-default-dockercfg-dwb92\"" Jul 06 23:53:33.410375 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.410344 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg"] Jul 06 23:53:33.555142 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.555110 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlnkt\" (UniqueName: \"kubernetes.io/projected/e8e8aad8-51b9-4159-807c-b9e03b0828cb-kube-api-access-zlnkt\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.555330 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.555154 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-workload-socket\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.555330 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.555182 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-data\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.555330 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.555247 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-workload-certs\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.555330 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.555296 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-podinfo\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.555559 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.555367 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istiod-ca-cert\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.555559 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.555410 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-envoy\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.555559 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.555437 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-token\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.555559 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.555485 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-credential-socket\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.656861 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.656771 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlnkt\" (UniqueName: \"kubernetes.io/projected/e8e8aad8-51b9-4159-807c-b9e03b0828cb-kube-api-access-zlnkt\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.656861 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.656828 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-workload-socket\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.656861 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.656853 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-data\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.657157 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.656878 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-workload-certs\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.657157 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.656909 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-podinfo\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.657157 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.656941 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istiod-ca-cert\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.657316 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.657193 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-envoy\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.657316 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.657244 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-token\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.657316 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.657299 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-data\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.657468 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.657314 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-workload-socket\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.657468 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.657300 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-credential-socket\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.657468 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.657425 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-workload-certs\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.657763 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.657706 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istiod-ca-cert\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.658055 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.657783 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-credential-socket\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.659533 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.659504 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-envoy\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.659893 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.659874 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-podinfo\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.665252 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.665231 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-token\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.665648 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.665624 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlnkt\" (UniqueName: \"kubernetes.io/projected/e8e8aad8-51b9-4159-807c-b9e03b0828cb-kube-api-access-zlnkt\") pod \"e2e-mt-5443d911-openshift-default-584758f679-hldrg\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.739855 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.739816 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:33.900119 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.900094 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg"] Jul 06 23:53:33.902434 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:53:33.902400 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e8aad8_51b9_4159_807c_b9e03b0828cb.slice/crio-5fee103a81726d6c468cd4a166cc3ed1911605402a3edc5bbea2b1226b1def17 WatchSource:0}: Error finding container 5fee103a81726d6c468cd4a166cc3ed1911605402a3edc5bbea2b1226b1def17: Status 404 returned error can't find the container with id 5fee103a81726d6c468cd4a166cc3ed1911605402a3edc5bbea2b1226b1def17 Jul 06 23:53:33.904617 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.904584 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:53:33.904703 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.904652 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:53:33.904703 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:33.904681 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:53:34.506731 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:34.506694 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" event={"ID":"e8e8aad8-51b9-4159-807c-b9e03b0828cb","Type":"ContainerStarted","Data":"7b17f4d2e4e568c7f8e9036b67be94f10454d3eaf586c6b8d53c06bfc0da4289"} Jul 06 23:53:34.507237 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:34.506766 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" event={"ID":"e8e8aad8-51b9-4159-807c-b9e03b0828cb","Type":"ContainerStarted","Data":"5fee103a81726d6c468cd4a166cc3ed1911605402a3edc5bbea2b1226b1def17"} Jul 06 23:53:34.527506 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:34.527451 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" podStartSLOduration=1.5274365479999998 podStartE2EDuration="1.527436548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:53:34.525377184 +0000 UTC m=+1725.239431753" watchObservedRunningTime="2026-07-06 23:53:34.527436548 +0000 UTC m=+1725.241491118" Jul 06 23:53:34.740191 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:34.740155 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:34.744932 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:34.744909 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:35.513136 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:35.513103 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:35.514155 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:35.514134 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:49.587958 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:49.587922 2571 generic.go:356] "Generic (PLEG): container finished" podID="e8e8aad8-51b9-4159-807c-b9e03b0828cb" containerID="7b17f4d2e4e568c7f8e9036b67be94f10454d3eaf586c6b8d53c06bfc0da4289" exitCode=0 Jul 06 23:53:49.588478 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:49.588003 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" event={"ID":"e8e8aad8-51b9-4159-807c-b9e03b0828cb","Type":"ContainerDied","Data":"7b17f4d2e4e568c7f8e9036b67be94f10454d3eaf586c6b8d53c06bfc0da4289"} Jul 06 23:53:49.588988 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:49.588958 2571 scope.go:117] "RemoveContainer" containerID="7b17f4d2e4e568c7f8e9036b67be94f10454d3eaf586c6b8d53c06bfc0da4289" Jul 06 23:53:49.591078 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:49.591036 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:53:49.591342 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:49.591312 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:53:49.591542 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:49.591517 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:53:50.513679 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:50.513640 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:50.595570 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:50.595534 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" event={"ID":"e8e8aad8-51b9-4159-807c-b9e03b0828cb","Type":"ContainerStarted","Data":"dfaa1c4ce86dd823781f89cc3927bd28b0f49071095ccbb7fd91d994090f1680"} Jul 06 23:53:50.740662 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:50.740621 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:50.741912 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:50.741884 2571 patch_prober.go:28] interesting pod/e2e-mt-5443d911-openshift-default-584758f679-hldrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 06 23:53:50.742034 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:50.741939 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" podUID="e8e8aad8-51b9-4159-807c-b9e03b0828cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 06 23:53:51.741025 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:51.740991 2571 patch_prober.go:28] interesting pod/e2e-mt-5443d911-openshift-default-584758f679-hldrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 06 23:53:51.741412 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:51.741054 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" podUID="e8e8aad8-51b9-4159-807c-b9e03b0828cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 06 23:53:52.740893 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:52.740861 2571 patch_prober.go:28] interesting pod/e2e-mt-5443d911-openshift-default-584758f679-hldrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 06 23:53:52.741061 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:52.740923 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" podUID="e8e8aad8-51b9-4159-807c-b9e03b0828cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 06 23:53:53.740605 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:53.740569 2571 patch_prober.go:28] interesting pod/e2e-mt-5443d911-openshift-default-584758f679-hldrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 06 23:53:53.740803 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:53.740622 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" podUID="e8e8aad8-51b9-4159-807c-b9e03b0828cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 06 23:53:54.744263 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:54.744229 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:54.744639 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:54.744484 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:53:54.745125 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:53:54.745106 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:54:49.838489 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:54:49.838463 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 06 23:54:49.859045 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:54:49.859021 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 06 23:55:18.256211 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:18.256179 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg"] Jul 06 23:55:18.256767 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:18.256400 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" podUID="e8e8aad8-51b9-4159-807c-b9e03b0828cb" containerName="istio-proxy" containerID="cri-o://dfaa1c4ce86dd823781f89cc3927bd28b0f49071095ccbb7fd91d994090f1680" gracePeriod=30 Jul 06 23:55:23.513610 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.513578 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:55:23.602994 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.602906 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-envoy\") pod \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " Jul 06 23:55:23.602994 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.602952 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istiod-ca-cert\") pod \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " Jul 06 23:55:23.602994 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.602989 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-data\") pod \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " Jul 06 23:55:23.603279 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.603024 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlnkt\" (UniqueName: \"kubernetes.io/projected/e8e8aad8-51b9-4159-807c-b9e03b0828cb-kube-api-access-zlnkt\") pod \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " Jul 06 23:55:23.603279 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.603049 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-workload-certs\") pod \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " Jul 06 23:55:23.603279 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.603081 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-workload-socket\") pod \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " Jul 06 23:55:23.603279 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.603114 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-credential-socket\") pod \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " Jul 06 23:55:23.603279 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.603155 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-podinfo\") pod \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " Jul 06 23:55:23.603279 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.603180 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-token\") pod \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\" (UID: \"e8e8aad8-51b9-4159-807c-b9e03b0828cb\") " Jul 06 23:55:23.603592 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.603407 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e8e8aad8-51b9-4159-807c-b9e03b0828cb" (UID: "e8e8aad8-51b9-4159-807c-b9e03b0828cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:55:23.603592 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.603452 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e8e8aad8-51b9-4159-807c-b9e03b0828cb" (UID: "e8e8aad8-51b9-4159-807c-b9e03b0828cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:55:23.603592 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.603544 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-workload-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:55:23.603592 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.603565 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-workload-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:55:23.603846 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.603646 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e8e8aad8-51b9-4159-807c-b9e03b0828cb" (UID: "e8e8aad8-51b9-4159-807c-b9e03b0828cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:55:23.603954 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.603928 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e8e8aad8-51b9-4159-807c-b9e03b0828cb" (UID: "e8e8aad8-51b9-4159-807c-b9e03b0828cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:55:23.604058 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.604012 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-data" (OuterVolumeSpecName: "istio-data") pod "e8e8aad8-51b9-4159-807c-b9e03b0828cb" (UID: "e8e8aad8-51b9-4159-807c-b9e03b0828cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:55:23.605481 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.605456 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e8e8aad8-51b9-4159-807c-b9e03b0828cb" (UID: "e8e8aad8-51b9-4159-807c-b9e03b0828cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:55:23.605819 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.605797 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e8aad8-51b9-4159-807c-b9e03b0828cb-kube-api-access-zlnkt" (OuterVolumeSpecName: "kube-api-access-zlnkt") pod "e8e8aad8-51b9-4159-807c-b9e03b0828cb" (UID: "e8e8aad8-51b9-4159-807c-b9e03b0828cb"). InnerVolumeSpecName "kube-api-access-zlnkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:55:23.606102 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.606081 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e8e8aad8-51b9-4159-807c-b9e03b0828cb" (UID: "e8e8aad8-51b9-4159-807c-b9e03b0828cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 23:55:23.606167 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.606082 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-token" (OuterVolumeSpecName: "istio-token") pod "e8e8aad8-51b9-4159-807c-b9e03b0828cb" (UID: "e8e8aad8-51b9-4159-807c-b9e03b0828cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:55:23.704677 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.704640 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-envoy\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:55:23.704677 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.704674 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istiod-ca-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:55:23.704677 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.704685 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-data\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:55:23.704934 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.704697 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlnkt\" (UniqueName: \"kubernetes.io/projected/e8e8aad8-51b9-4159-807c-b9e03b0828cb-kube-api-access-zlnkt\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:55:23.704934 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.704709 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8aad8-51b9-4159-807c-b9e03b0828cb-credential-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:55:23.704934 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.704736 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-podinfo\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:55:23.704934 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:23.704749 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e8aad8-51b9-4159-807c-b9e03b0828cb-istio-token\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:55:24.083192 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:24.083159 2571 generic.go:356] "Generic (PLEG): container finished" podID="e8e8aad8-51b9-4159-807c-b9e03b0828cb" containerID="dfaa1c4ce86dd823781f89cc3927bd28b0f49071095ccbb7fd91d994090f1680" exitCode=0 Jul 06 23:55:24.083364 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:24.083208 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" Jul 06 23:55:24.083364 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:24.083214 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" event={"ID":"e8e8aad8-51b9-4159-807c-b9e03b0828cb","Type":"ContainerDied","Data":"dfaa1c4ce86dd823781f89cc3927bd28b0f49071095ccbb7fd91d994090f1680"} Jul 06 23:55:24.083364 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:24.083267 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg" event={"ID":"e8e8aad8-51b9-4159-807c-b9e03b0828cb","Type":"ContainerDied","Data":"5fee103a81726d6c468cd4a166cc3ed1911605402a3edc5bbea2b1226b1def17"} Jul 06 23:55:24.083364 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:24.083291 2571 scope.go:117] "RemoveContainer" containerID="dfaa1c4ce86dd823781f89cc3927bd28b0f49071095ccbb7fd91d994090f1680" Jul 06 23:55:24.092559 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:24.092542 2571 scope.go:117] "RemoveContainer" containerID="7b17f4d2e4e568c7f8e9036b67be94f10454d3eaf586c6b8d53c06bfc0da4289" Jul 06 23:55:24.105992 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:24.102772 2571 scope.go:117] "RemoveContainer" containerID="dfaa1c4ce86dd823781f89cc3927bd28b0f49071095ccbb7fd91d994090f1680" Jul 06 23:55:24.106942 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:55:24.106787 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfaa1c4ce86dd823781f89cc3927bd28b0f49071095ccbb7fd91d994090f1680\": container with ID starting with dfaa1c4ce86dd823781f89cc3927bd28b0f49071095ccbb7fd91d994090f1680 not found: ID does not exist" containerID="dfaa1c4ce86dd823781f89cc3927bd28b0f49071095ccbb7fd91d994090f1680" Jul 06 23:55:24.106942 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:24.106848 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfaa1c4ce86dd823781f89cc3927bd28b0f49071095ccbb7fd91d994090f1680"} err="failed to get container status \"dfaa1c4ce86dd823781f89cc3927bd28b0f49071095ccbb7fd91d994090f1680\": rpc error: code = NotFound desc = could not find container \"dfaa1c4ce86dd823781f89cc3927bd28b0f49071095ccbb7fd91d994090f1680\": container with ID starting with dfaa1c4ce86dd823781f89cc3927bd28b0f49071095ccbb7fd91d994090f1680 not found: ID does not exist" Jul 06 23:55:24.106942 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:24.106886 2571 scope.go:117] "RemoveContainer" containerID="7b17f4d2e4e568c7f8e9036b67be94f10454d3eaf586c6b8d53c06bfc0da4289" Jul 06 23:55:24.107693 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:55:24.107662 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b17f4d2e4e568c7f8e9036b67be94f10454d3eaf586c6b8d53c06bfc0da4289\": container with ID starting with 7b17f4d2e4e568c7f8e9036b67be94f10454d3eaf586c6b8d53c06bfc0da4289 not found: ID does not exist" containerID="7b17f4d2e4e568c7f8e9036b67be94f10454d3eaf586c6b8d53c06bfc0da4289" Jul 06 23:55:24.107809 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:24.107707 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b17f4d2e4e568c7f8e9036b67be94f10454d3eaf586c6b8d53c06bfc0da4289"} err="failed to get container status \"7b17f4d2e4e568c7f8e9036b67be94f10454d3eaf586c6b8d53c06bfc0da4289\": rpc error: code = NotFound desc = could not find container \"7b17f4d2e4e568c7f8e9036b67be94f10454d3eaf586c6b8d53c06bfc0da4289\": container with ID starting with 7b17f4d2e4e568c7f8e9036b67be94f10454d3eaf586c6b8d53c06bfc0da4289 not found: ID does not exist" Jul 06 23:55:24.116117 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:24.116090 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg"] Jul 06 23:55:24.121379 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:24.121352 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5443d911-openshift-default-584758f679-hldrg"] Jul 06 23:55:25.785950 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:55:25.785919 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8e8aad8-51b9-4159-807c-b9e03b0828cb" path="/var/lib/kubelet/pods/e8e8aad8-51b9-4159-807c-b9e03b0828cb/volumes" Jul 06 23:56:12.981607 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:12.981534 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh"] Jul 06 23:56:12.982043 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:12.981882 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e8aad8-51b9-4159-807c-b9e03b0828cb" containerName="istio-proxy" Jul 06 23:56:12.982043 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:12.981894 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e8aad8-51b9-4159-807c-b9e03b0828cb" containerName="istio-proxy" Jul 06 23:56:12.982043 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:12.981921 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6f8ba3-8c39-4614-ba25-d377223fb5cb" containerName="istio-proxy" Jul 06 23:56:12.982043 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:12.981926 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6f8ba3-8c39-4614-ba25-d377223fb5cb" containerName="istio-proxy" Jul 06 23:56:12.982043 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:12.981977 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e8aad8-51b9-4159-807c-b9e03b0828cb" containerName="istio-proxy" Jul 06 23:56:12.982043 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:12.981987 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e8aad8-51b9-4159-807c-b9e03b0828cb" containerName="istio-proxy" Jul 06 23:56:12.985074 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:12.985054 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:12.988434 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:12.988402 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8c6acb-openshift-default-dockercfg-2k479\"" Jul 06 23:56:13.000749 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.000702 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh"] Jul 06 23:56:13.115251 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.115188 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-workload-socket\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.115476 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.115256 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-workload-certs\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.115476 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.115289 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-credential-socket\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.115476 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.115350 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istiod-ca-cert\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.115476 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.115385 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-envoy\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.115476 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.115440 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-podinfo\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.115476 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.115466 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-data\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.115842 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.115491 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9nz\" (UniqueName: \"kubernetes.io/projected/24c10a72-cfd4-4196-ace9-f8cf5c31400f-kube-api-access-df9nz\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.115842 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.115518 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-token\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.216706 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.216673 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-envoy\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.216929 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.216733 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-podinfo\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.216929 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.216876 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-data\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.216929 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.216920 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9nz\" (UniqueName: \"kubernetes.io/projected/24c10a72-cfd4-4196-ace9-f8cf5c31400f-kube-api-access-df9nz\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.217099 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.216949 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-token\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.217099 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.217003 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-workload-socket\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.217393 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.217372 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-workload-certs\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.217488 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.217414 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-credential-socket\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.217547 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.217497 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istiod-ca-cert\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.217601 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.217311 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-workload-socket\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.218440 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.217238 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-data\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.218440 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.217842 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-workload-certs\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.218440 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.217889 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-credential-socket\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.218440 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.218396 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istiod-ca-cert\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.219279 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.219247 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-envoy\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.219396 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.219341 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-podinfo\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.225003 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.224982 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-token\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.225385 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.225363 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9nz\" (UniqueName: \"kubernetes.io/projected/24c10a72-cfd4-4196-ace9-f8cf5c31400f-kube-api-access-df9nz\") pod \"e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.295817 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.295740 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:13.436386 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.436364 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh"] Jul 06 23:56:13.438201 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:56:13.438169 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c10a72_cfd4_4196_ace9_f8cf5c31400f.slice/crio-cd43ff123a5e047213b86c6727f51a21a0d1b22248530eb101221a1d93d38a29 WatchSource:0}: Error finding container cd43ff123a5e047213b86c6727f51a21a0d1b22248530eb101221a1d93d38a29: Status 404 returned error can't find the container with id cd43ff123a5e047213b86c6727f51a21a0d1b22248530eb101221a1d93d38a29 Jul 06 23:56:13.440327 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.440290 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:56:13.440406 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.440371 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:56:13.440457 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:13.440421 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:56:14.343490 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:14.343451 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" event={"ID":"24c10a72-cfd4-4196-ace9-f8cf5c31400f","Type":"ContainerStarted","Data":"85ee566b3294c95600f07e148a3bed36091f6725dc4ad5ea1ead03bf447174da"} Jul 06 23:56:14.343877 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:14.343510 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" event={"ID":"24c10a72-cfd4-4196-ace9-f8cf5c31400f","Type":"ContainerStarted","Data":"cd43ff123a5e047213b86c6727f51a21a0d1b22248530eb101221a1d93d38a29"} Jul 06 23:56:14.366897 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:14.366848 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" podStartSLOduration=2.366829751 podStartE2EDuration="2.366829751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:14.365822964 +0000 UTC m=+1885.079877535" watchObservedRunningTime="2026-07-06 23:56:14.366829751 +0000 UTC m=+1885.080884321" Jul 06 23:56:15.296268 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:15.296227 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:15.301172 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:15.301147 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:15.348967 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:15.348935 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:15.349835 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:15.349816 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:21.478793 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.478744 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d"] Jul 06 23:56:21.479336 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.479320 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e8aad8-51b9-4159-807c-b9e03b0828cb" containerName="istio-proxy" Jul 06 23:56:21.479386 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.479339 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e8aad8-51b9-4159-807c-b9e03b0828cb" containerName="istio-proxy" Jul 06 23:56:21.482724 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.482701 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.485283 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.485265 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4820fd-openshift-default-dockercfg-xfxmg\"" Jul 06 23:56:21.492407 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.492384 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d"] Jul 06 23:56:21.596063 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.595989 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069687f7-e6e2-4d20-8583-31e134452f9b-istiod-ca-cert\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.596571 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.596523 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr8q5\" (UniqueName: \"kubernetes.io/projected/069687f7-e6e2-4d20-8583-31e134452f9b-kube-api-access-mr8q5\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.596711 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.596647 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-workload-socket\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.596711 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.596674 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069687f7-e6e2-4d20-8583-31e134452f9b-istio-podinfo\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.596906 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.596758 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-credential-socket\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.596906 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.596796 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-istio-data\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.596906 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.596853 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-workload-certs\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.597075 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.596879 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-istio-envoy\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.597160 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.597099 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069687f7-e6e2-4d20-8583-31e134452f9b-istio-token\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.698171 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.698138 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-workload-certs\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.698171 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.698173 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-istio-envoy\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.698382 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.698193 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069687f7-e6e2-4d20-8583-31e134452f9b-istio-token\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.698382 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.698239 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069687f7-e6e2-4d20-8583-31e134452f9b-istiod-ca-cert\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.698382 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.698259 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr8q5\" (UniqueName: \"kubernetes.io/projected/069687f7-e6e2-4d20-8583-31e134452f9b-kube-api-access-mr8q5\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.698382 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.698333 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-workload-socket\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.698586 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.698469 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069687f7-e6e2-4d20-8583-31e134452f9b-istio-podinfo\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.698586 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.698507 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-credential-socket\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.698586 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.698538 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-istio-data\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.698778 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.698614 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-workload-certs\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.698778 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.698680 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-workload-socket\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.698946 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.698916 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-credential-socket\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.698946 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.698932 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-istio-data\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.699103 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.698973 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069687f7-e6e2-4d20-8583-31e134452f9b-istiod-ca-cert\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.700979 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.700959 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-istio-envoy\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.701108 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.701087 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069687f7-e6e2-4d20-8583-31e134452f9b-istio-podinfo\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.705808 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.705788 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069687f7-e6e2-4d20-8583-31e134452f9b-istio-token\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.706066 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.706047 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr8q5\" (UniqueName: \"kubernetes.io/projected/069687f7-e6e2-4d20-8583-31e134452f9b-kube-api-access-mr8q5\") pod \"e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.794775 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.794695 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:21.937671 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.937593 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d"] Jul 06 23:56:21.939921 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:56:21.939893 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069687f7_e6e2_4d20_8583_31e134452f9b.slice/crio-20e08d6fd018701608f9dad6addfef8947a053be46caba123c60a67882db438a WatchSource:0}: Error finding container 20e08d6fd018701608f9dad6addfef8947a053be46caba123c60a67882db438a: Status 404 returned error can't find the container with id 20e08d6fd018701608f9dad6addfef8947a053be46caba123c60a67882db438a Jul 06 23:56:21.942100 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.942069 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:56:21.942181 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.942131 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:56:21.942181 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:21.942160 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:56:22.386817 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:22.386774 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" event={"ID":"069687f7-e6e2-4d20-8583-31e134452f9b","Type":"ContainerStarted","Data":"4eda2ce9c5a242e4a9d9103762777e7228ae80a59ce9af1ad1717a7372534be6"} Jul 06 23:56:22.387027 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:22.386840 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" event={"ID":"069687f7-e6e2-4d20-8583-31e134452f9b","Type":"ContainerStarted","Data":"20e08d6fd018701608f9dad6addfef8947a053be46caba123c60a67882db438a"} Jul 06 23:56:22.407409 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:22.407360 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" podStartSLOduration=1.407344111 podStartE2EDuration="1.407344111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:56:22.405610314 +0000 UTC m=+1893.119664885" watchObservedRunningTime="2026-07-06 23:56:22.407344111 +0000 UTC m=+1893.121398682" Jul 06 23:56:22.795883 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:22.795849 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:22.801035 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:22.801009 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:23.392652 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:23.392618 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:23.393758 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:23.393737 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:30.497651 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:30.497608 2571 patch_prober.go:28] interesting pod/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": read tcp 10.134.0.2:36610->10.134.0.56:15021: read: connection reset by peer" start-of-body= Jul 06 23:56:30.498241 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:30.497688 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" podUID="24c10a72-cfd4-4196-ace9-f8cf5c31400f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": read tcp 10.134.0.2:36610->10.134.0.56:15021: read: connection reset by peer" Jul 06 23:56:31.444438 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:31.444395 2571 generic.go:356] "Generic (PLEG): container finished" podID="24c10a72-cfd4-4196-ace9-f8cf5c31400f" containerID="85ee566b3294c95600f07e148a3bed36091f6725dc4ad5ea1ead03bf447174da" exitCode=0 Jul 06 23:56:31.444630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:31.444470 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" event={"ID":"24c10a72-cfd4-4196-ace9-f8cf5c31400f","Type":"ContainerDied","Data":"85ee566b3294c95600f07e148a3bed36091f6725dc4ad5ea1ead03bf447174da"} Jul 06 23:56:31.444925 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:31.444911 2571 scope.go:117] "RemoveContainer" containerID="85ee566b3294c95600f07e148a3bed36091f6725dc4ad5ea1ead03bf447174da" Jul 06 23:56:31.445928 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:31.445896 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:56:31.445987 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:31.445966 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:56:31.446028 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:31.445998 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:56:32.456951 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:32.456906 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" event={"ID":"24c10a72-cfd4-4196-ace9-f8cf5c31400f","Type":"ContainerStarted","Data":"80ad5f8d4a02e6088c9d67d63ddf10d1c63a0b010628810c7dca87c681c45bc0"} Jul 06 23:56:33.296280 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:33.296238 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:33.296591 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:33.296569 2571 patch_prober.go:28] interesting pod/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 06 23:56:33.296646 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:33.296628 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" podUID="24c10a72-cfd4-4196-ace9-f8cf5c31400f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 06 23:56:34.296945 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:34.296913 2571 patch_prober.go:28] interesting pod/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 06 23:56:34.297313 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:34.296967 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" podUID="24c10a72-cfd4-4196-ace9-f8cf5c31400f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 06 23:56:35.296565 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:35.296532 2571 patch_prober.go:28] interesting pod/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 06 23:56:35.296821 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:35.296592 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" podUID="24c10a72-cfd4-4196-ace9-f8cf5c31400f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 06 23:56:36.296683 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:36.296643 2571 patch_prober.go:28] interesting pod/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 06 23:56:36.297190 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:36.296704 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" podUID="24c10a72-cfd4-4196-ace9-f8cf5c31400f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 06 23:56:37.299585 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:37.299559 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:37.300014 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:37.299845 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:37.300544 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:37.300527 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:56:37.487551 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:37.487519 2571 generic.go:356] "Generic (PLEG): container finished" podID="069687f7-e6e2-4d20-8583-31e134452f9b" containerID="4eda2ce9c5a242e4a9d9103762777e7228ae80a59ce9af1ad1717a7372534be6" exitCode=0 Jul 06 23:56:37.487752 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:37.487597 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" event={"ID":"069687f7-e6e2-4d20-8583-31e134452f9b","Type":"ContainerDied","Data":"4eda2ce9c5a242e4a9d9103762777e7228ae80a59ce9af1ad1717a7372534be6"} Jul 06 23:56:37.488188 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:37.488167 2571 scope.go:117] "RemoveContainer" containerID="4eda2ce9c5a242e4a9d9103762777e7228ae80a59ce9af1ad1717a7372534be6" Jul 06 23:56:37.489233 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:37.489206 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:56:37.489336 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:37.489271 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:56:37.489336 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:37.489301 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:56:38.393395 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:38.393353 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:38.494750 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:38.494687 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" event={"ID":"069687f7-e6e2-4d20-8583-31e134452f9b","Type":"ContainerStarted","Data":"515f142a35f9c775d4b61a87c40d2b6fcb64a8c4b921d75675b5c174bc26cec8"} Jul 06 23:56:38.795195 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:38.795155 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:38.796308 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:38.796280 2571 patch_prober.go:28] interesting pod/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 06 23:56:38.796415 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:38.796365 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" podUID="069687f7-e6e2-4d20-8583-31e134452f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 06 23:56:39.795211 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:39.795170 2571 patch_prober.go:28] interesting pod/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 06 23:56:39.795609 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:39.795243 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" podUID="069687f7-e6e2-4d20-8583-31e134452f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 06 23:56:40.795389 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:40.795352 2571 patch_prober.go:28] interesting pod/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 06 23:56:40.795829 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:40.795411 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" podUID="069687f7-e6e2-4d20-8583-31e134452f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 06 23:56:41.796052 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:41.796018 2571 patch_prober.go:28] interesting pod/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 06 23:56:41.796465 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:41.796070 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" podUID="069687f7-e6e2-4d20-8583-31e134452f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 06 23:56:42.799156 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:42.799125 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:42.799562 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:42.799384 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:56:42.800039 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:56:42.800024 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:57:21.872836 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:21.872799 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6"] Jul 06 23:57:21.876128 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:21.876109 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:21.879659 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:21.879635 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4820fd\"/\"test-model-4820fd-kserve-self-signed-certs\"" Jul 06 23:57:21.879806 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:21.879640 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4820fd\"/\"kube-root-ca.crt\"" Jul 06 23:57:21.879806 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:21.879644 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4820fd\"/\"openshift-service-ca.crt\"" Jul 06 23:57:21.879806 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:21.879693 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4820fd\"/\"default-dockercfg-htlsn\"" Jul 06 23:57:21.885346 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:21.885325 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6"] Jul 06 23:57:21.946139 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:21.946105 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-home\") pod \"test-model-4820fd-kserve-558cbf49b5-2dcq6\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:21.946335 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:21.946156 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn9gn\" (UniqueName: \"kubernetes.io/projected/fe4443d0-bb89-4484-ace1-1f04bfb00170-kube-api-access-sn9gn\") pod \"test-model-4820fd-kserve-558cbf49b5-2dcq6\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:21.946335 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:21.946191 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-dshm\") pod \"test-model-4820fd-kserve-558cbf49b5-2dcq6\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:21.946335 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:21.946220 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe4443d0-bb89-4484-ace1-1f04bfb00170-tls-certs\") pod \"test-model-4820fd-kserve-558cbf49b5-2dcq6\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:21.946335 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:21.946288 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-tmp-dir\") pod \"test-model-4820fd-kserve-558cbf49b5-2dcq6\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:21.946335 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:21.946329 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-model-cache\") pod \"test-model-4820fd-kserve-558cbf49b5-2dcq6\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:22.047247 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:22.047216 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn9gn\" (UniqueName: \"kubernetes.io/projected/fe4443d0-bb89-4484-ace1-1f04bfb00170-kube-api-access-sn9gn\") pod \"test-model-4820fd-kserve-558cbf49b5-2dcq6\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:22.047446 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:22.047255 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-dshm\") pod \"test-model-4820fd-kserve-558cbf49b5-2dcq6\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:22.047446 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:22.047290 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe4443d0-bb89-4484-ace1-1f04bfb00170-tls-certs\") pod \"test-model-4820fd-kserve-558cbf49b5-2dcq6\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:22.047446 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:22.047350 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-tmp-dir\") pod \"test-model-4820fd-kserve-558cbf49b5-2dcq6\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:22.047619 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:22.047488 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-model-cache\") pod \"test-model-4820fd-kserve-558cbf49b5-2dcq6\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:22.047619 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:22.047581 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-home\") pod \"test-model-4820fd-kserve-558cbf49b5-2dcq6\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:22.047796 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:22.047778 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-tmp-dir\") pod \"test-model-4820fd-kserve-558cbf49b5-2dcq6\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:22.047880 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:22.047860 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-model-cache\") pod \"test-model-4820fd-kserve-558cbf49b5-2dcq6\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:22.047927 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:22.047905 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-home\") pod \"test-model-4820fd-kserve-558cbf49b5-2dcq6\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:22.049702 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:22.049677 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-dshm\") pod \"test-model-4820fd-kserve-558cbf49b5-2dcq6\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:22.050019 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:22.050002 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe4443d0-bb89-4484-ace1-1f04bfb00170-tls-certs\") pod \"test-model-4820fd-kserve-558cbf49b5-2dcq6\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:22.056105 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:22.056086 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn9gn\" (UniqueName: \"kubernetes.io/projected/fe4443d0-bb89-4484-ace1-1f04bfb00170-kube-api-access-sn9gn\") pod \"test-model-4820fd-kserve-558cbf49b5-2dcq6\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:22.186347 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:22.186316 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:22.325710 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:22.325685 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6"] Jul 06 23:57:22.327775 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:57:22.327745 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe4443d0_bb89_4484_ace1_1f04bfb00170.slice/crio-21aa38c7d820679ff98561d518f66650431dcee38999f68118bfa8b5200897de WatchSource:0}: Error finding container 21aa38c7d820679ff98561d518f66650431dcee38999f68118bfa8b5200897de: Status 404 returned error can't find the container with id 21aa38c7d820679ff98561d518f66650431dcee38999f68118bfa8b5200897de Jul 06 23:57:22.733209 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:22.733165 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" event={"ID":"fe4443d0-bb89-4484-ace1-1f04bfb00170","Type":"ContainerStarted","Data":"8c8b0e13c2c6c7751d3bab2732977a03b125de57cdd1f8c52487d6e0ddce6fa4"} Jul 06 23:57:22.733209 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:22.733212 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" event={"ID":"fe4443d0-bb89-4484-ace1-1f04bfb00170","Type":"ContainerStarted","Data":"21aa38c7d820679ff98561d518f66650431dcee38999f68118bfa8b5200897de"} Jul 06 23:57:22.749784 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:22.749708 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" podStartSLOduration=1.7496941590000001 podStartE2EDuration="1.749694159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:57:22.748057461 +0000 UTC m=+1953.462112045" watchObservedRunningTime="2026-07-06 23:57:22.749694159 +0000 UTC m=+1953.463748728" Jul 06 23:57:32.189147 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:32.189087 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:32.189681 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:32.189214 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:32.210122 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:32.210085 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:32.798435 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:32.798402 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:57:48.905753 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:57:48.905625 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6"] Jul 06 23:58:03.926936 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:03.926897 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" podUID="fe4443d0-bb89-4484-ace1-1f04bfb00170" containerName="main" containerID="cri-o://8c8b0e13c2c6c7751d3bab2732977a03b125de57cdd1f8c52487d6e0ddce6fa4" gracePeriod=45 Jul 06 23:58:04.173980 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.173953 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:58:04.238817 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.238735 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-home\") pod \"fe4443d0-bb89-4484-ace1-1f04bfb00170\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " Jul 06 23:58:04.238817 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.238788 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-model-cache\") pod \"fe4443d0-bb89-4484-ace1-1f04bfb00170\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " Jul 06 23:58:04.238817 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.238816 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-tmp-dir\") pod \"fe4443d0-bb89-4484-ace1-1f04bfb00170\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " Jul 06 23:58:04.239069 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.238848 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sn9gn\" (UniqueName: \"kubernetes.io/projected/fe4443d0-bb89-4484-ace1-1f04bfb00170-kube-api-access-sn9gn\") pod \"fe4443d0-bb89-4484-ace1-1f04bfb00170\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " Jul 06 23:58:04.239069 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.238904 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-dshm\") pod \"fe4443d0-bb89-4484-ace1-1f04bfb00170\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " Jul 06 23:58:04.239069 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.238906 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-home" (OuterVolumeSpecName: "home") pod "fe4443d0-bb89-4484-ace1-1f04bfb00170" (UID: "fe4443d0-bb89-4484-ace1-1f04bfb00170"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:58:04.239069 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.238956 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe4443d0-bb89-4484-ace1-1f04bfb00170-tls-certs\") pod \"fe4443d0-bb89-4484-ace1-1f04bfb00170\" (UID: \"fe4443d0-bb89-4484-ace1-1f04bfb00170\") " Jul 06 23:58:04.239283 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.239085 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe4443d0-bb89-4484-ace1-1f04bfb00170" (UID: "fe4443d0-bb89-4484-ace1-1f04bfb00170"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:58:04.239283 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.239100 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe4443d0-bb89-4484-ace1-1f04bfb00170" (UID: "fe4443d0-bb89-4484-ace1-1f04bfb00170"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:58:04.239283 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.239269 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-home\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:04.239403 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.239288 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-model-cache\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:04.239403 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.239303 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-tmp-dir\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:04.241331 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.241301 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-dshm" (OuterVolumeSpecName: "dshm") pod "fe4443d0-bb89-4484-ace1-1f04bfb00170" (UID: "fe4443d0-bb89-4484-ace1-1f04bfb00170"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:58:04.241331 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.241318 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4443d0-bb89-4484-ace1-1f04bfb00170-kube-api-access-sn9gn" (OuterVolumeSpecName: "kube-api-access-sn9gn") pod "fe4443d0-bb89-4484-ace1-1f04bfb00170" (UID: "fe4443d0-bb89-4484-ace1-1f04bfb00170"). InnerVolumeSpecName "kube-api-access-sn9gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:58:04.241516 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.241335 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe4443d0-bb89-4484-ace1-1f04bfb00170-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe4443d0-bb89-4484-ace1-1f04bfb00170" (UID: "fe4443d0-bb89-4484-ace1-1f04bfb00170"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:58:04.340202 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.340169 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn9gn\" (UniqueName: \"kubernetes.io/projected/fe4443d0-bb89-4484-ace1-1f04bfb00170-kube-api-access-sn9gn\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:04.340202 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.340195 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe4443d0-bb89-4484-ace1-1f04bfb00170-dshm\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:04.340202 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.340204 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe4443d0-bb89-4484-ace1-1f04bfb00170-tls-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:04.963972 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.963935 2571 generic.go:356] "Generic (PLEG): container finished" podID="fe4443d0-bb89-4484-ace1-1f04bfb00170" containerID="8c8b0e13c2c6c7751d3bab2732977a03b125de57cdd1f8c52487d6e0ddce6fa4" exitCode=0 Jul 06 23:58:04.964425 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.963995 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" Jul 06 23:58:04.964425 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.964020 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" event={"ID":"fe4443d0-bb89-4484-ace1-1f04bfb00170","Type":"ContainerDied","Data":"8c8b0e13c2c6c7751d3bab2732977a03b125de57cdd1f8c52487d6e0ddce6fa4"} Jul 06 23:58:04.964425 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.964076 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6" event={"ID":"fe4443d0-bb89-4484-ace1-1f04bfb00170","Type":"ContainerDied","Data":"21aa38c7d820679ff98561d518f66650431dcee38999f68118bfa8b5200897de"} Jul 06 23:58:04.964425 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.964103 2571 scope.go:117] "RemoveContainer" containerID="8c8b0e13c2c6c7751d3bab2732977a03b125de57cdd1f8c52487d6e0ddce6fa4" Jul 06 23:58:04.984653 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.984628 2571 scope.go:117] "RemoveContainer" containerID="8c8b0e13c2c6c7751d3bab2732977a03b125de57cdd1f8c52487d6e0ddce6fa4" Jul 06 23:58:04.985065 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:58:04.985039 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c8b0e13c2c6c7751d3bab2732977a03b125de57cdd1f8c52487d6e0ddce6fa4\": container with ID starting with 8c8b0e13c2c6c7751d3bab2732977a03b125de57cdd1f8c52487d6e0ddce6fa4 not found: ID does not exist" containerID="8c8b0e13c2c6c7751d3bab2732977a03b125de57cdd1f8c52487d6e0ddce6fa4" Jul 06 23:58:04.985157 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.985077 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8b0e13c2c6c7751d3bab2732977a03b125de57cdd1f8c52487d6e0ddce6fa4"} err="failed to get container status \"8c8b0e13c2c6c7751d3bab2732977a03b125de57cdd1f8c52487d6e0ddce6fa4\": rpc error: code = NotFound desc = could not find container \"8c8b0e13c2c6c7751d3bab2732977a03b125de57cdd1f8c52487d6e0ddce6fa4\": container with ID starting with 8c8b0e13c2c6c7751d3bab2732977a03b125de57cdd1f8c52487d6e0ddce6fa4 not found: ID does not exist" Jul 06 23:58:04.990301 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:58:04.990262 2571 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe4443d0_bb89_4484_ace1_1f04bfb00170.slice/crio-21aa38c7d820679ff98561d518f66650431dcee38999f68118bfa8b5200897de\": RecentStats: unable to find data in memory cache]" Jul 06 23:58:04.990841 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.990815 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6"] Jul 06 23:58:04.992969 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:04.992946 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4820fd/test-model-4820fd-kserve-558cbf49b5-2dcq6"] Jul 06 23:58:05.786242 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:05.786207 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe4443d0-bb89-4484-ace1-1f04bfb00170" path="/var/lib/kubelet/pods/fe4443d0-bb89-4484-ace1-1f04bfb00170/volumes" Jul 06 23:58:31.776473 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:31.776437 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh"] Jul 06 23:58:31.777059 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:31.776658 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" podUID="24c10a72-cfd4-4196-ace9-f8cf5c31400f" containerName="istio-proxy" containerID="cri-o://80ad5f8d4a02e6088c9d67d63ddf10d1c63a0b010628810c7dca87c681c45bc0" gracePeriod=30 Jul 06 23:58:37.024263 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.024238 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:58:37.137380 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.137294 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-token\") pod \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " Jul 06 23:58:37.137380 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.137325 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-podinfo\") pod \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " Jul 06 23:58:37.137380 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.137359 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istiod-ca-cert\") pod \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " Jul 06 23:58:37.137676 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.137402 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-workload-certs\") pod \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " Jul 06 23:58:37.137676 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.137427 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-envoy\") pod \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " Jul 06 23:58:37.137676 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.137466 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-workload-socket\") pod \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " Jul 06 23:58:37.137676 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.137591 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-df9nz\" (UniqueName: \"kubernetes.io/projected/24c10a72-cfd4-4196-ace9-f8cf5c31400f-kube-api-access-df9nz\") pod \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " Jul 06 23:58:37.137676 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.137666 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-credential-socket\") pod \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " Jul 06 23:58:37.137958 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.137729 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-data\") pod \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\" (UID: \"24c10a72-cfd4-4196-ace9-f8cf5c31400f\") " Jul 06 23:58:37.138032 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.137982 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "24c10a72-cfd4-4196-ace9-f8cf5c31400f" (UID: "24c10a72-cfd4-4196-ace9-f8cf5c31400f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:58:37.138219 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.138140 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "24c10a72-cfd4-4196-ace9-f8cf5c31400f" (UID: "24c10a72-cfd4-4196-ace9-f8cf5c31400f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:58:37.138219 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.138224 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "24c10a72-cfd4-4196-ace9-f8cf5c31400f" (UID: "24c10a72-cfd4-4196-ace9-f8cf5c31400f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:58:37.138474 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.138261 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istiod-ca-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:37.138600 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.138578 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "24c10a72-cfd4-4196-ace9-f8cf5c31400f" (UID: "24c10a72-cfd4-4196-ace9-f8cf5c31400f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:58:37.139136 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.139108 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-data" (OuterVolumeSpecName: "istio-data") pod "24c10a72-cfd4-4196-ace9-f8cf5c31400f" (UID: "24c10a72-cfd4-4196-ace9-f8cf5c31400f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:58:37.139576 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.139547 2571 generic.go:356] "Generic (PLEG): container finished" podID="24c10a72-cfd4-4196-ace9-f8cf5c31400f" containerID="80ad5f8d4a02e6088c9d67d63ddf10d1c63a0b010628810c7dca87c681c45bc0" exitCode=0 Jul 06 23:58:37.139688 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.139608 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" event={"ID":"24c10a72-cfd4-4196-ace9-f8cf5c31400f","Type":"ContainerDied","Data":"80ad5f8d4a02e6088c9d67d63ddf10d1c63a0b010628810c7dca87c681c45bc0"} Jul 06 23:58:37.139688 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.139659 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" event={"ID":"24c10a72-cfd4-4196-ace9-f8cf5c31400f","Type":"ContainerDied","Data":"cd43ff123a5e047213b86c6727f51a21a0d1b22248530eb101221a1d93d38a29"} Jul 06 23:58:37.139871 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.139740 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh" Jul 06 23:58:37.139871 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.139689 2571 scope.go:117] "RemoveContainer" containerID="80ad5f8d4a02e6088c9d67d63ddf10d1c63a0b010628810c7dca87c681c45bc0" Jul 06 23:58:37.140756 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.140696 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "24c10a72-cfd4-4196-ace9-f8cf5c31400f" (UID: "24c10a72-cfd4-4196-ace9-f8cf5c31400f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 23:58:37.140854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.140760 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "24c10a72-cfd4-4196-ace9-f8cf5c31400f" (UID: "24c10a72-cfd4-4196-ace9-f8cf5c31400f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:58:37.140854 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.140798 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-token" (OuterVolumeSpecName: "istio-token") pod "24c10a72-cfd4-4196-ace9-f8cf5c31400f" (UID: "24c10a72-cfd4-4196-ace9-f8cf5c31400f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:58:37.140951 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.140851 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c10a72-cfd4-4196-ace9-f8cf5c31400f-kube-api-access-df9nz" (OuterVolumeSpecName: "kube-api-access-df9nz") pod "24c10a72-cfd4-4196-ace9-f8cf5c31400f" (UID: "24c10a72-cfd4-4196-ace9-f8cf5c31400f"). InnerVolumeSpecName "kube-api-access-df9nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:58:37.158247 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.158223 2571 scope.go:117] "RemoveContainer" containerID="85ee566b3294c95600f07e148a3bed36091f6725dc4ad5ea1ead03bf447174da" Jul 06 23:58:37.166588 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.166571 2571 scope.go:117] "RemoveContainer" containerID="80ad5f8d4a02e6088c9d67d63ddf10d1c63a0b010628810c7dca87c681c45bc0" Jul 06 23:58:37.166902 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:58:37.166882 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ad5f8d4a02e6088c9d67d63ddf10d1c63a0b010628810c7dca87c681c45bc0\": container with ID starting with 80ad5f8d4a02e6088c9d67d63ddf10d1c63a0b010628810c7dca87c681c45bc0 not found: ID does not exist" containerID="80ad5f8d4a02e6088c9d67d63ddf10d1c63a0b010628810c7dca87c681c45bc0" Jul 06 23:58:37.166973 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.166912 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ad5f8d4a02e6088c9d67d63ddf10d1c63a0b010628810c7dca87c681c45bc0"} err="failed to get container status \"80ad5f8d4a02e6088c9d67d63ddf10d1c63a0b010628810c7dca87c681c45bc0\": rpc error: code = NotFound desc = could not find container \"80ad5f8d4a02e6088c9d67d63ddf10d1c63a0b010628810c7dca87c681c45bc0\": container with ID starting with 80ad5f8d4a02e6088c9d67d63ddf10d1c63a0b010628810c7dca87c681c45bc0 not found: ID does not exist" Jul 06 23:58:37.166973 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.166931 2571 scope.go:117] "RemoveContainer" containerID="85ee566b3294c95600f07e148a3bed36091f6725dc4ad5ea1ead03bf447174da" Jul 06 23:58:37.167196 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:58:37.167182 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ee566b3294c95600f07e148a3bed36091f6725dc4ad5ea1ead03bf447174da\": container with ID starting with 85ee566b3294c95600f07e148a3bed36091f6725dc4ad5ea1ead03bf447174da not found: ID does not exist" containerID="85ee566b3294c95600f07e148a3bed36091f6725dc4ad5ea1ead03bf447174da" Jul 06 23:58:37.167247 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.167200 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ee566b3294c95600f07e148a3bed36091f6725dc4ad5ea1ead03bf447174da"} err="failed to get container status \"85ee566b3294c95600f07e148a3bed36091f6725dc4ad5ea1ead03bf447174da\": rpc error: code = NotFound desc = could not find container \"85ee566b3294c95600f07e148a3bed36091f6725dc4ad5ea1ead03bf447174da\": container with ID starting with 85ee566b3294c95600f07e148a3bed36091f6725dc4ad5ea1ead03bf447174da not found: ID does not exist" Jul 06 23:58:37.239750 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.239673 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-credential-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:37.239750 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.239709 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-data\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:37.239964 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.239764 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-token\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:37.239964 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.239772 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-podinfo\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:37.239964 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.239782 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-workload-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:37.239964 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.239790 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-istio-envoy\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:37.239964 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.239798 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c10a72-cfd4-4196-ace9-f8cf5c31400f-workload-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:37.239964 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.239807 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df9nz\" (UniqueName: \"kubernetes.io/projected/24c10a72-cfd4-4196-ace9-f8cf5c31400f-kube-api-access-df9nz\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:37.462294 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.462264 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh"] Jul 06 23:58:37.466655 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.466632 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8c6acb-openshift-default-c8c889479-v84vh"] Jul 06 23:58:37.785946 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:37.785869 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24c10a72-cfd4-4196-ace9-f8cf5c31400f" path="/var/lib/kubelet/pods/24c10a72-cfd4-4196-ace9-f8cf5c31400f/volumes" Jul 06 23:58:38.685190 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:38.685154 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d"] Jul 06 23:58:38.685643 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:38.685435 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" podUID="069687f7-e6e2-4d20-8583-31e134452f9b" containerName="istio-proxy" containerID="cri-o://515f142a35f9c775d4b61a87c40d2b6fcb64a8c4b921d75675b5c174bc26cec8" gracePeriod=30 Jul 06 23:58:39.614080 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.614049 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh"] Jul 06 23:58:39.614441 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.614427 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24c10a72-cfd4-4196-ace9-f8cf5c31400f" containerName="istio-proxy" Jul 06 23:58:39.614505 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.614442 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="24c10a72-cfd4-4196-ace9-f8cf5c31400f" containerName="istio-proxy" Jul 06 23:58:39.614505 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.614453 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe4443d0-bb89-4484-ace1-1f04bfb00170" containerName="main" Jul 06 23:58:39.614505 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.614458 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe4443d0-bb89-4484-ace1-1f04bfb00170" containerName="main" Jul 06 23:58:39.614651 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.614513 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe4443d0-bb89-4484-ace1-1f04bfb00170" containerName="main" Jul 06 23:58:39.614651 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.614520 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="24c10a72-cfd4-4196-ace9-f8cf5c31400f" containerName="istio-proxy" Jul 06 23:58:39.619107 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.619087 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.621327 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.621304 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-87c0b0-openshift-default-dockercfg-qpw6f\"" Jul 06 23:58:39.629139 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.629115 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh"] Jul 06 23:58:39.761226 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.761196 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-podinfo\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.761226 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.761229 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-token\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.761669 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.761267 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vxzk\" (UniqueName: \"kubernetes.io/projected/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-kube-api-access-8vxzk\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.761669 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.761336 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-workload-socket\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.761669 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.761389 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-workload-certs\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.761669 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.761408 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-credential-socket\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.761669 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.761431 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istiod-ca-cert\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.761669 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.761451 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-envoy\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.761669 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.761494 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-data\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.862114 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.862082 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-workload-certs\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.862294 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.862129 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-credential-socket\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.862294 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.862171 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istiod-ca-cert\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.862294 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.862203 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-envoy\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.862294 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.862242 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-data\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.862294 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.862287 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-podinfo\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.862578 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.862315 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-token\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.862578 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.862371 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vxzk\" (UniqueName: \"kubernetes.io/projected/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-kube-api-access-8vxzk\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.862578 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.862412 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-workload-socket\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.862578 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.862524 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-workload-certs\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.862932 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.862897 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-credential-socket\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.863228 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.862967 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-data\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.863228 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.863042 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-workload-socket\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.863228 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.863164 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istiod-ca-cert\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.864753 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.864678 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-envoy\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.865006 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.864987 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-podinfo\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.870349 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.870322 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-token\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.870917 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.870894 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vxzk\" (UniqueName: \"kubernetes.io/projected/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-kube-api-access-8vxzk\") pod \"e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:39.932738 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:39.932687 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:40.083139 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:40.083113 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh"] Jul 06 23:58:40.085373 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:58:40.085338 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839bf18d_bec2_4bfa_83f5_f8e1f4fa2b50.slice/crio-3361dc90ab18aa8bcf63936651f59a1bdfe668b21a90ceadcc17edcface98068 WatchSource:0}: Error finding container 3361dc90ab18aa8bcf63936651f59a1bdfe668b21a90ceadcc17edcface98068: Status 404 returned error can't find the container with id 3361dc90ab18aa8bcf63936651f59a1bdfe668b21a90ceadcc17edcface98068 Jul 06 23:58:40.087542 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:40.087502 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:58:40.087647 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:40.087590 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:58:40.087647 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:40.087629 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:58:40.159040 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:40.158974 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" event={"ID":"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50","Type":"ContainerStarted","Data":"f57c456bee779bda806efe624552ee29b1f1f2262c8f2a0ad897bb1bd58e3c15"} Jul 06 23:58:40.159040 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:40.159022 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" event={"ID":"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50","Type":"ContainerStarted","Data":"3361dc90ab18aa8bcf63936651f59a1bdfe668b21a90ceadcc17edcface98068"} Jul 06 23:58:40.181037 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:40.180979 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" podStartSLOduration=1.180955556 podStartE2EDuration="1.180955556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:58:40.178161641 +0000 UTC m=+2030.892216213" watchObservedRunningTime="2026-07-06 23:58:40.180955556 +0000 UTC m=+2030.895010128" Jul 06 23:58:40.933540 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:40.933502 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:41.938457 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:41.938425 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:42.169217 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:42.169185 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:42.170234 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:42.170209 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:42.801610 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:42.801577 2571 patch_prober.go:28] interesting pod/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 23:58:42.801811 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:42.801636 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" podUID="069687f7-e6e2-4d20-8583-31e134452f9b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 23:58:43.939235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:43.939210 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:58:44.000828 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.000739 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-workload-socket\") pod \"069687f7-e6e2-4d20-8583-31e134452f9b\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " Jul 06 23:58:44.000828 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.000785 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069687f7-e6e2-4d20-8583-31e134452f9b-istiod-ca-cert\") pod \"069687f7-e6e2-4d20-8583-31e134452f9b\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " Jul 06 23:58:44.000828 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.000821 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mr8q5\" (UniqueName: \"kubernetes.io/projected/069687f7-e6e2-4d20-8583-31e134452f9b-kube-api-access-mr8q5\") pod \"069687f7-e6e2-4d20-8583-31e134452f9b\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " Jul 06 23:58:44.001109 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.000848 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069687f7-e6e2-4d20-8583-31e134452f9b-istio-podinfo\") pod \"069687f7-e6e2-4d20-8583-31e134452f9b\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " Jul 06 23:58:44.001109 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.000881 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-workload-certs\") pod \"069687f7-e6e2-4d20-8583-31e134452f9b\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " Jul 06 23:58:44.001109 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.000902 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-istio-envoy\") pod \"069687f7-e6e2-4d20-8583-31e134452f9b\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " Jul 06 23:58:44.001109 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.000919 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069687f7-e6e2-4d20-8583-31e134452f9b-istio-token\") pod \"069687f7-e6e2-4d20-8583-31e134452f9b\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " Jul 06 23:58:44.001109 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.000949 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-istio-data\") pod \"069687f7-e6e2-4d20-8583-31e134452f9b\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " Jul 06 23:58:44.001109 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.000995 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-credential-socket\") pod \"069687f7-e6e2-4d20-8583-31e134452f9b\" (UID: \"069687f7-e6e2-4d20-8583-31e134452f9b\") " Jul 06 23:58:44.001109 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.001053 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "069687f7-e6e2-4d20-8583-31e134452f9b" (UID: "069687f7-e6e2-4d20-8583-31e134452f9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:58:44.001436 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.001246 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "069687f7-e6e2-4d20-8583-31e134452f9b" (UID: "069687f7-e6e2-4d20-8583-31e134452f9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:58:44.001436 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.001256 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-workload-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:44.001436 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.001259 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/069687f7-e6e2-4d20-8583-31e134452f9b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "069687f7-e6e2-4d20-8583-31e134452f9b" (UID: "069687f7-e6e2-4d20-8583-31e134452f9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:58:44.001605 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.001453 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "069687f7-e6e2-4d20-8583-31e134452f9b" (UID: "069687f7-e6e2-4d20-8583-31e134452f9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:58:44.001856 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.001832 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-istio-data" (OuterVolumeSpecName: "istio-data") pod "069687f7-e6e2-4d20-8583-31e134452f9b" (UID: "069687f7-e6e2-4d20-8583-31e134452f9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:58:44.003322 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.003292 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069687f7-e6e2-4d20-8583-31e134452f9b-kube-api-access-mr8q5" (OuterVolumeSpecName: "kube-api-access-mr8q5") pod "069687f7-e6e2-4d20-8583-31e134452f9b" (UID: "069687f7-e6e2-4d20-8583-31e134452f9b"). InnerVolumeSpecName "kube-api-access-mr8q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:58:44.003322 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.003304 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069687f7-e6e2-4d20-8583-31e134452f9b-istio-token" (OuterVolumeSpecName: "istio-token") pod "069687f7-e6e2-4d20-8583-31e134452f9b" (UID: "069687f7-e6e2-4d20-8583-31e134452f9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:58:44.003710 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.003692 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "069687f7-e6e2-4d20-8583-31e134452f9b" (UID: "069687f7-e6e2-4d20-8583-31e134452f9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:58:44.003791 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.003709 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/069687f7-e6e2-4d20-8583-31e134452f9b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "069687f7-e6e2-4d20-8583-31e134452f9b" (UID: "069687f7-e6e2-4d20-8583-31e134452f9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 23:58:44.102686 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.102645 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-credential-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:44.102686 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.102679 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069687f7-e6e2-4d20-8583-31e134452f9b-istiod-ca-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:44.102686 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.102691 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr8q5\" (UniqueName: \"kubernetes.io/projected/069687f7-e6e2-4d20-8583-31e134452f9b-kube-api-access-mr8q5\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:44.102966 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.102702 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069687f7-e6e2-4d20-8583-31e134452f9b-istio-podinfo\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:44.102966 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.102711 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-workload-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:44.102966 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.102750 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-istio-envoy\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:44.102966 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.102758 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069687f7-e6e2-4d20-8583-31e134452f9b-istio-token\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:44.102966 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.102765 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069687f7-e6e2-4d20-8583-31e134452f9b-istio-data\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:58:44.182177 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.182144 2571 generic.go:356] "Generic (PLEG): container finished" podID="069687f7-e6e2-4d20-8583-31e134452f9b" containerID="515f142a35f9c775d4b61a87c40d2b6fcb64a8c4b921d75675b5c174bc26cec8" exitCode=0 Jul 06 23:58:44.182384 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.182236 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" Jul 06 23:58:44.182384 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.182247 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" event={"ID":"069687f7-e6e2-4d20-8583-31e134452f9b","Type":"ContainerDied","Data":"515f142a35f9c775d4b61a87c40d2b6fcb64a8c4b921d75675b5c174bc26cec8"} Jul 06 23:58:44.182384 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.182303 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d" event={"ID":"069687f7-e6e2-4d20-8583-31e134452f9b","Type":"ContainerDied","Data":"20e08d6fd018701608f9dad6addfef8947a053be46caba123c60a67882db438a"} Jul 06 23:58:44.182384 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.182326 2571 scope.go:117] "RemoveContainer" containerID="515f142a35f9c775d4b61a87c40d2b6fcb64a8c4b921d75675b5c174bc26cec8" Jul 06 23:58:44.192588 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.192569 2571 scope.go:117] "RemoveContainer" containerID="4eda2ce9c5a242e4a9d9103762777e7228ae80a59ce9af1ad1717a7372534be6" Jul 06 23:58:44.201963 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.201944 2571 scope.go:117] "RemoveContainer" containerID="515f142a35f9c775d4b61a87c40d2b6fcb64a8c4b921d75675b5c174bc26cec8" Jul 06 23:58:44.202267 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:58:44.202248 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515f142a35f9c775d4b61a87c40d2b6fcb64a8c4b921d75675b5c174bc26cec8\": container with ID starting with 515f142a35f9c775d4b61a87c40d2b6fcb64a8c4b921d75675b5c174bc26cec8 not found: ID does not exist" containerID="515f142a35f9c775d4b61a87c40d2b6fcb64a8c4b921d75675b5c174bc26cec8" Jul 06 23:58:44.202352 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.202280 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515f142a35f9c775d4b61a87c40d2b6fcb64a8c4b921d75675b5c174bc26cec8"} err="failed to get container status \"515f142a35f9c775d4b61a87c40d2b6fcb64a8c4b921d75675b5c174bc26cec8\": rpc error: code = NotFound desc = could not find container \"515f142a35f9c775d4b61a87c40d2b6fcb64a8c4b921d75675b5c174bc26cec8\": container with ID starting with 515f142a35f9c775d4b61a87c40d2b6fcb64a8c4b921d75675b5c174bc26cec8 not found: ID does not exist" Jul 06 23:58:44.202352 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.202304 2571 scope.go:117] "RemoveContainer" containerID="4eda2ce9c5a242e4a9d9103762777e7228ae80a59ce9af1ad1717a7372534be6" Jul 06 23:58:44.202561 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:58:44.202544 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eda2ce9c5a242e4a9d9103762777e7228ae80a59ce9af1ad1717a7372534be6\": container with ID starting with 4eda2ce9c5a242e4a9d9103762777e7228ae80a59ce9af1ad1717a7372534be6 not found: ID does not exist" containerID="4eda2ce9c5a242e4a9d9103762777e7228ae80a59ce9af1ad1717a7372534be6" Jul 06 23:58:44.202604 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.202567 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eda2ce9c5a242e4a9d9103762777e7228ae80a59ce9af1ad1717a7372534be6"} err="failed to get container status \"4eda2ce9c5a242e4a9d9103762777e7228ae80a59ce9af1ad1717a7372534be6\": rpc error: code = NotFound desc = could not find container \"4eda2ce9c5a242e4a9d9103762777e7228ae80a59ce9af1ad1717a7372534be6\": container with ID starting with 4eda2ce9c5a242e4a9d9103762777e7228ae80a59ce9af1ad1717a7372534be6 not found: ID does not exist" Jul 06 23:58:44.208503 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.208473 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d"] Jul 06 23:58:44.215456 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:44.215433 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4820fd-openshift-default-686c8b8667-bvf4d"] Jul 06 23:58:45.786282 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:45.786250 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="069687f7-e6e2-4d20-8583-31e134452f9b" path="/var/lib/kubelet/pods/069687f7-e6e2-4d20-8583-31e134452f9b/volumes" Jul 06 23:58:47.390954 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.390919 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd"] Jul 06 23:58:47.391335 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.391307 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="069687f7-e6e2-4d20-8583-31e134452f9b" containerName="istio-proxy" Jul 06 23:58:47.391335 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.391319 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="069687f7-e6e2-4d20-8583-31e134452f9b" containerName="istio-proxy" Jul 06 23:58:47.391335 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.391327 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24c10a72-cfd4-4196-ace9-f8cf5c31400f" containerName="istio-proxy" Jul 06 23:58:47.391335 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.391333 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="24c10a72-cfd4-4196-ace9-f8cf5c31400f" containerName="istio-proxy" Jul 06 23:58:47.391467 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.391345 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="069687f7-e6e2-4d20-8583-31e134452f9b" containerName="istio-proxy" Jul 06 23:58:47.391467 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.391355 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="069687f7-e6e2-4d20-8583-31e134452f9b" containerName="istio-proxy" Jul 06 23:58:47.391467 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.391410 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="069687f7-e6e2-4d20-8583-31e134452f9b" containerName="istio-proxy" Jul 06 23:58:47.391467 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.391421 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="24c10a72-cfd4-4196-ace9-f8cf5c31400f" containerName="istio-proxy" Jul 06 23:58:47.391467 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.391428 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="069687f7-e6e2-4d20-8583-31e134452f9b" containerName="istio-proxy" Jul 06 23:58:47.395964 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.395937 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.398366 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.398347 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-68ab62-openshift-default-dockercfg-bjxgv\"" Jul 06 23:58:47.407917 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.407887 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd"] Jul 06 23:58:47.431863 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.431824 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d685979-b511-488d-9df5-6d688167a770-istiod-ca-cert\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.431863 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.431865 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx6qh\" (UniqueName: \"kubernetes.io/projected/5d685979-b511-488d-9df5-6d688167a770-kube-api-access-mx6qh\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.432134 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.431899 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-istio-data\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.432134 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.431942 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-credential-socket\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.432134 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.431983 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-workload-socket\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.432134 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.432026 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-workload-certs\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.432134 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.432076 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-istio-envoy\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.432134 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.432121 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d685979-b511-488d-9df5-6d688167a770-istio-token\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.432369 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.432156 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d685979-b511-488d-9df5-6d688167a770-istio-podinfo\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.533210 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.532792 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d685979-b511-488d-9df5-6d688167a770-istio-podinfo\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.533210 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.532872 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d685979-b511-488d-9df5-6d688167a770-istiod-ca-cert\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.533210 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.532903 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx6qh\" (UniqueName: \"kubernetes.io/projected/5d685979-b511-488d-9df5-6d688167a770-kube-api-access-mx6qh\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.533210 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.532937 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-istio-data\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.533210 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.532989 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-credential-socket\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.533210 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.533022 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-workload-socket\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.533210 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.533072 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-workload-certs\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.533210 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.533101 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-istio-envoy\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.533210 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.533149 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d685979-b511-488d-9df5-6d688167a770-istio-token\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.534025 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.533491 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-workload-socket\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.534025 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.533690 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d685979-b511-488d-9df5-6d688167a770-istiod-ca-cert\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.534025 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.533800 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-workload-certs\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.534191 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.534037 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-credential-socket\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.534514 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.534488 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-istio-data\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.536219 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.536193 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d685979-b511-488d-9df5-6d688167a770-istio-podinfo\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.536551 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.536531 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-istio-envoy\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.541247 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.541221 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx6qh\" (UniqueName: \"kubernetes.io/projected/5d685979-b511-488d-9df5-6d688167a770-kube-api-access-mx6qh\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.541601 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.541579 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d685979-b511-488d-9df5-6d688167a770-istio-token\") pod \"e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.708140 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.708098 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:47.872782 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.872687 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd"] Jul 06 23:58:47.875588 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:58:47.875551 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d685979_b511_488d_9df5_6d688167a770.slice/crio-aaabd0be831e9bfd54793f33107a40ffc57ed4ec5f3537d0b142cabe867dd528 WatchSource:0}: Error finding container aaabd0be831e9bfd54793f33107a40ffc57ed4ec5f3537d0b142cabe867dd528: Status 404 returned error can't find the container with id aaabd0be831e9bfd54793f33107a40ffc57ed4ec5f3537d0b142cabe867dd528 Jul 06 23:58:47.878498 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.878466 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:58:47.878578 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.878531 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:58:47.878578 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:47.878561 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:58:48.205826 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:48.205795 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" event={"ID":"5d685979-b511-488d-9df5-6d688167a770","Type":"ContainerStarted","Data":"14935df0f57c4e2c09674693039687ab62dded246f93e7c45f13c20ad258f4c1"} Jul 06 23:58:48.206012 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:48.205851 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" event={"ID":"5d685979-b511-488d-9df5-6d688167a770","Type":"ContainerStarted","Data":"aaabd0be831e9bfd54793f33107a40ffc57ed4ec5f3537d0b142cabe867dd528"} Jul 06 23:58:48.230513 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:48.230445 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" podStartSLOduration=1.230423265 podStartE2EDuration="1.230423265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:58:48.226278192 +0000 UTC m=+2038.940332753" watchObservedRunningTime="2026-07-06 23:58:48.230423265 +0000 UTC m=+2038.944477834" Jul 06 23:58:48.709022 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:48.708980 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:48.714566 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:48.714543 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:49.212622 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:49.212587 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:49.213543 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:49.213526 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:58:57.171160 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:57.171128 2571 patch_prober.go:28] interesting pod/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 06 23:58:57.171592 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:57.171195 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" podUID="839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 06 23:58:57.265363 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:57.265330 2571 generic.go:356] "Generic (PLEG): container finished" podID="839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" containerID="f57c456bee779bda806efe624552ee29b1f1f2262c8f2a0ad897bb1bd58e3c15" exitCode=0 Jul 06 23:58:57.265527 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:57.265389 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" event={"ID":"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50","Type":"ContainerDied","Data":"f57c456bee779bda806efe624552ee29b1f1f2262c8f2a0ad897bb1bd58e3c15"} Jul 06 23:58:57.265782 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:57.265767 2571 scope.go:117] "RemoveContainer" containerID="f57c456bee779bda806efe624552ee29b1f1f2262c8f2a0ad897bb1bd58e3c15" Jul 06 23:58:57.266787 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:57.266756 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:58:57.266893 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:57.266844 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:58:57.266961 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:57.266889 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:58:58.273597 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:58.273563 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" event={"ID":"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50","Type":"ContainerStarted","Data":"4cf48239ccbb93c94bcead6479c0de70ed6b97b7e504e245a5130af132cbcf52"} Jul 06 23:58:58.933810 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:58.933320 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:58:58.933810 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:58.933464 2571 patch_prober.go:28] interesting pod/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 06 23:58:58.933810 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:58.933509 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" podUID="839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 06 23:58:59.933526 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:59.933472 2571 patch_prober.go:28] interesting pod/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 06 23:58:59.934076 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:58:59.933551 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" podUID="839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 06 23:59:00.933867 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:00.933820 2571 patch_prober.go:28] interesting pod/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 06 23:59:00.934394 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:00.933889 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" podUID="839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 06 23:59:01.933699 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:01.933224 2571 patch_prober.go:28] interesting pod/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 06 23:59:01.933699 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:01.933308 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" podUID="839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 06 23:59:02.937353 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:02.937318 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:59:02.937813 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:02.937592 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:59:02.938292 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:02.938274 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:59:05.253934 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:05.253897 2571 patch_prober.go:28] interesting pod/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 23:59:05.254356 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:05.253966 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" podUID="5d685979-b511-488d-9df5-6d688167a770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 23:59:06.317919 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:06.317879 2571 generic.go:356] "Generic (PLEG): container finished" podID="5d685979-b511-488d-9df5-6d688167a770" containerID="14935df0f57c4e2c09674693039687ab62dded246f93e7c45f13c20ad258f4c1" exitCode=0 Jul 06 23:59:06.318461 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:06.317933 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" event={"ID":"5d685979-b511-488d-9df5-6d688167a770","Type":"ContainerDied","Data":"14935df0f57c4e2c09674693039687ab62dded246f93e7c45f13c20ad258f4c1"} Jul 06 23:59:06.318461 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:06.318302 2571 scope.go:117] "RemoveContainer" containerID="14935df0f57c4e2c09674693039687ab62dded246f93e7c45f13c20ad258f4c1" Jul 06 23:59:06.319367 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:06.319334 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:59:06.319490 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:06.319402 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:59:06.319490 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:06.319434 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:59:07.330643 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:07.330596 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" event={"ID":"5d685979-b511-488d-9df5-6d688167a770","Type":"ContainerStarted","Data":"15dc281ce1eb2d922a4b4b950633a8eee90b0062aed286eef65b09e2c75864e9"} Jul 06 23:59:07.709573 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:07.709468 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:59:07.709573 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:07.709553 2571 patch_prober.go:28] interesting pod/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 06 23:59:07.709888 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:07.709604 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" podUID="5d685979-b511-488d-9df5-6d688167a770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 06 23:59:08.709587 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:08.709544 2571 patch_prober.go:28] interesting pod/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 06 23:59:08.710214 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:08.710178 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" podUID="5d685979-b511-488d-9df5-6d688167a770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 06 23:59:09.709167 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:09.709117 2571 patch_prober.go:28] interesting pod/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 06 23:59:09.709973 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:09.709938 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" podUID="5d685979-b511-488d-9df5-6d688167a770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 06 23:59:09.787786 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:09.787755 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh"] Jul 06 23:59:09.787979 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:09.787948 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" podUID="839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" containerName="istio-proxy" containerID="cri-o://4cf48239ccbb93c94bcead6479c0de70ed6b97b7e504e245a5130af132cbcf52" gracePeriod=30 Jul 06 23:59:10.709000 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:10.708906 2571 patch_prober.go:28] interesting pod/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 06 23:59:10.709000 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:10.708974 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" podUID="5d685979-b511-488d-9df5-6d688167a770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 06 23:59:11.713303 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:11.713270 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:59:11.713824 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:11.713552 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:59:11.714245 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:11.714229 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:59:15.040706 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.040678 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:59:15.205700 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.205666 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-workload-socket\") pod \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " Jul 06 23:59:15.205929 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.205748 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istiod-ca-cert\") pod \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " Jul 06 23:59:15.205929 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.205765 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-workload-certs\") pod \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " Jul 06 23:59:15.205929 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.205800 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-podinfo\") pod \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " Jul 06 23:59:15.205929 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.205818 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vxzk\" (UniqueName: \"kubernetes.io/projected/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-kube-api-access-8vxzk\") pod \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " Jul 06 23:59:15.205929 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.205840 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-token\") pod \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " Jul 06 23:59:15.205929 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.205872 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-envoy\") pod \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " Jul 06 23:59:15.205929 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.205919 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-data\") pod \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " Jul 06 23:59:15.206312 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.205992 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-credential-socket\") pod \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\" (UID: \"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50\") " Jul 06 23:59:15.206312 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.205982 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" (UID: "839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:59:15.206312 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.206136 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" (UID: "839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:59:15.206312 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.206143 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" (UID: "839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:59:15.206675 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.206336 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-workload-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:59:15.206675 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.206360 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istiod-ca-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:59:15.206675 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.206375 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-workload-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:59:15.206675 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.206401 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" (UID: "839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:59:15.207196 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.207163 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-data" (OuterVolumeSpecName: "istio-data") pod "839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" (UID: "839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:59:15.208217 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.208194 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" (UID: "839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 23:59:15.208306 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.208218 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-kube-api-access-8vxzk" (OuterVolumeSpecName: "kube-api-access-8vxzk") pod "839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" (UID: "839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50"). InnerVolumeSpecName "kube-api-access-8vxzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:59:15.208306 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.208231 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" (UID: "839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:59:15.208418 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.208359 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-token" (OuterVolumeSpecName: "istio-token") pod "839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" (UID: "839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:59:15.307624 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.307587 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-envoy\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:59:15.307624 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.307618 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-data\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:59:15.307624 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.307629 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-credential-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:59:15.307624 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.307637 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-podinfo\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:59:15.307935 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.307646 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vxzk\" (UniqueName: \"kubernetes.io/projected/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-kube-api-access-8vxzk\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:59:15.307935 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.307656 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50-istio-token\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:59:15.377460 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.377420 2571 generic.go:356] "Generic (PLEG): container finished" podID="839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" containerID="4cf48239ccbb93c94bcead6479c0de70ed6b97b7e504e245a5130af132cbcf52" exitCode=0 Jul 06 23:59:15.377630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.377473 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" Jul 06 23:59:15.377630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.377507 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" event={"ID":"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50","Type":"ContainerDied","Data":"4cf48239ccbb93c94bcead6479c0de70ed6b97b7e504e245a5130af132cbcf52"} Jul 06 23:59:15.377630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.377559 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh" event={"ID":"839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50","Type":"ContainerDied","Data":"3361dc90ab18aa8bcf63936651f59a1bdfe668b21a90ceadcc17edcface98068"} Jul 06 23:59:15.377630 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.377582 2571 scope.go:117] "RemoveContainer" containerID="4cf48239ccbb93c94bcead6479c0de70ed6b97b7e504e245a5130af132cbcf52" Jul 06 23:59:15.387082 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.387065 2571 scope.go:117] "RemoveContainer" containerID="f57c456bee779bda806efe624552ee29b1f1f2262c8f2a0ad897bb1bd58e3c15" Jul 06 23:59:15.395968 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.395928 2571 scope.go:117] "RemoveContainer" containerID="4cf48239ccbb93c94bcead6479c0de70ed6b97b7e504e245a5130af132cbcf52" Jul 06 23:59:15.396248 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:59:15.396227 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf48239ccbb93c94bcead6479c0de70ed6b97b7e504e245a5130af132cbcf52\": container with ID starting with 4cf48239ccbb93c94bcead6479c0de70ed6b97b7e504e245a5130af132cbcf52 not found: ID does not exist" containerID="4cf48239ccbb93c94bcead6479c0de70ed6b97b7e504e245a5130af132cbcf52" Jul 06 23:59:15.396337 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.396257 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf48239ccbb93c94bcead6479c0de70ed6b97b7e504e245a5130af132cbcf52"} err="failed to get container status \"4cf48239ccbb93c94bcead6479c0de70ed6b97b7e504e245a5130af132cbcf52\": rpc error: code = NotFound desc = could not find container \"4cf48239ccbb93c94bcead6479c0de70ed6b97b7e504e245a5130af132cbcf52\": container with ID starting with 4cf48239ccbb93c94bcead6479c0de70ed6b97b7e504e245a5130af132cbcf52 not found: ID does not exist" Jul 06 23:59:15.396337 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.396278 2571 scope.go:117] "RemoveContainer" containerID="f57c456bee779bda806efe624552ee29b1f1f2262c8f2a0ad897bb1bd58e3c15" Jul 06 23:59:15.397007 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:59:15.396667 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57c456bee779bda806efe624552ee29b1f1f2262c8f2a0ad897bb1bd58e3c15\": container with ID starting with f57c456bee779bda806efe624552ee29b1f1f2262c8f2a0ad897bb1bd58e3c15 not found: ID does not exist" containerID="f57c456bee779bda806efe624552ee29b1f1f2262c8f2a0ad897bb1bd58e3c15" Jul 06 23:59:15.397007 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.396800 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57c456bee779bda806efe624552ee29b1f1f2262c8f2a0ad897bb1bd58e3c15"} err="failed to get container status \"f57c456bee779bda806efe624552ee29b1f1f2262c8f2a0ad897bb1bd58e3c15\": rpc error: code = NotFound desc = could not find container \"f57c456bee779bda806efe624552ee29b1f1f2262c8f2a0ad897bb1bd58e3c15\": container with ID starting with f57c456bee779bda806efe624552ee29b1f1f2262c8f2a0ad897bb1bd58e3c15 not found: ID does not exist" Jul 06 23:59:15.398890 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.398868 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh"] Jul 06 23:59:15.402606 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.402586 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-87c0b0-openshift-default-556986955f-bdkfh"] Jul 06 23:59:15.786121 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:15.786085 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" path="/var/lib/kubelet/pods/839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50/volumes" Jul 06 23:59:16.456128 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:16.456086 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd"] Jul 06 23:59:16.456568 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:16.456326 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" podUID="5d685979-b511-488d-9df5-6d688167a770" containerName="istio-proxy" containerID="cri-o://15dc281ce1eb2d922a4b4b950633a8eee90b0062aed286eef65b09e2c75864e9" gracePeriod=30 Jul 06 23:59:17.346017 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.345811 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5"] Jul 06 23:59:17.347653 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.346660 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" containerName="istio-proxy" Jul 06 23:59:17.347653 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.346686 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" containerName="istio-proxy" Jul 06 23:59:17.347653 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.346767 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" containerName="istio-proxy" Jul 06 23:59:17.347653 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.346778 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" containerName="istio-proxy" Jul 06 23:59:17.347653 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.346965 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" containerName="istio-proxy" Jul 06 23:59:17.352641 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.352608 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.355144 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.355118 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-9ea6e9-openshift-default-dockercfg-glpbv\"" Jul 06 23:59:17.356540 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.356516 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5"] Jul 06 23:59:17.527873 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.527836 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istiod-ca-cert\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.527873 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.527879 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-workload-certs\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.528374 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.527905 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-data\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.528374 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.528006 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-envoy\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.528374 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.528080 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-credential-socket\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.528374 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.528112 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrstw\" (UniqueName: \"kubernetes.io/projected/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-kube-api-access-jrstw\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.528374 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.528165 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-workload-socket\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.528374 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.528213 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-podinfo\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.528374 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.528246 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-token\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.629713 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.629605 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-token\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.629713 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.629657 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istiod-ca-cert\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.629713 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.629683 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-workload-certs\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.629713 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.629706 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-data\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.630057 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.629867 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-envoy\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.630057 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.629932 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-credential-socket\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.630057 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.629961 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrstw\" (UniqueName: \"kubernetes.io/projected/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-kube-api-access-jrstw\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.630057 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.630046 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-workload-socket\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.630275 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.630095 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-podinfo\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.630275 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.630143 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-workload-certs\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.630275 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.630222 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-data\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.630440 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.630339 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-credential-socket\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.630516 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.630491 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-workload-socket\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.630568 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.630511 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istiod-ca-cert\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.632271 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.632237 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-envoy\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.632601 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.632581 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-podinfo\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.637884 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.637854 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-token\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.637980 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.637923 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrstw\" (UniqueName: \"kubernetes.io/projected/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-kube-api-access-jrstw\") pod \"e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.665425 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.665375 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:17.812154 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.812122 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5"] Jul 06 23:59:17.813973 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:59:17.813944 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca59f7c_5f3f_4353_a44c_e923cf1842fe.slice/crio-80921957cf04bdc191afa138d0e530cb9096a96a88bf8bfb57a023390af28412 WatchSource:0}: Error finding container 80921957cf04bdc191afa138d0e530cb9096a96a88bf8bfb57a023390af28412: Status 404 returned error can't find the container with id 80921957cf04bdc191afa138d0e530cb9096a96a88bf8bfb57a023390af28412 Jul 06 23:59:17.816610 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.816571 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:59:17.816688 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.816665 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:59:17.816757 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:17.816705 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:59:18.398036 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:18.397998 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" event={"ID":"9ca59f7c-5f3f-4353-a44c-e923cf1842fe","Type":"ContainerStarted","Data":"2558ee74c959fd63885790e339631ae6ce45f1e2516e09d03504757f8d286305"} Jul 06 23:59:18.398216 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:18.398051 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" event={"ID":"9ca59f7c-5f3f-4353-a44c-e923cf1842fe","Type":"ContainerStarted","Data":"80921957cf04bdc191afa138d0e530cb9096a96a88bf8bfb57a023390af28412"} Jul 06 23:59:18.418946 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:18.418898 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" podStartSLOduration=1.4188829809999999 podStartE2EDuration="1.418882981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:59:18.41654658 +0000 UTC m=+2069.130601152" watchObservedRunningTime="2026-07-06 23:59:18.418882981 +0000 UTC m=+2069.132937550" Jul 06 23:59:18.666518 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:18.666493 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:18.671319 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:18.671299 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:19.403227 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:19.403197 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:19.404083 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:19.404059 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:21.723265 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.723209 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:59:21.867834 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.867806 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-istio-data\") pod \"5d685979-b511-488d-9df5-6d688167a770\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " Jul 06 23:59:21.868024 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.867841 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-workload-socket\") pod \"5d685979-b511-488d-9df5-6d688167a770\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " Jul 06 23:59:21.868024 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.867892 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d685979-b511-488d-9df5-6d688167a770-istiod-ca-cert\") pod \"5d685979-b511-488d-9df5-6d688167a770\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " Jul 06 23:59:21.868024 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.867912 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-workload-certs\") pod \"5d685979-b511-488d-9df5-6d688167a770\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " Jul 06 23:59:21.868024 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.867953 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d685979-b511-488d-9df5-6d688167a770-istio-token\") pod \"5d685979-b511-488d-9df5-6d688167a770\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " Jul 06 23:59:21.868024 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.867986 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-istio-envoy\") pod \"5d685979-b511-488d-9df5-6d688167a770\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " Jul 06 23:59:21.868302 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.868082 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d685979-b511-488d-9df5-6d688167a770-istio-podinfo\") pod \"5d685979-b511-488d-9df5-6d688167a770\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " Jul 06 23:59:21.868302 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.868116 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-credential-socket\") pod \"5d685979-b511-488d-9df5-6d688167a770\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " Jul 06 23:59:21.868302 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.868168 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx6qh\" (UniqueName: \"kubernetes.io/projected/5d685979-b511-488d-9df5-6d688167a770-kube-api-access-mx6qh\") pod \"5d685979-b511-488d-9df5-6d688167a770\" (UID: \"5d685979-b511-488d-9df5-6d688167a770\") " Jul 06 23:59:21.868302 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.868222 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5d685979-b511-488d-9df5-6d688167a770" (UID: "5d685979-b511-488d-9df5-6d688167a770"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:59:21.868302 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.868235 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5d685979-b511-488d-9df5-6d688167a770" (UID: "5d685979-b511-488d-9df5-6d688167a770"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:59:21.868562 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.868448 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5d685979-b511-488d-9df5-6d688167a770" (UID: "5d685979-b511-488d-9df5-6d688167a770"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:59:21.868562 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.868462 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-workload-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:59:21.868562 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.868480 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-workload-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:59:21.868562 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.868478 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d685979-b511-488d-9df5-6d688167a770-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5d685979-b511-488d-9df5-6d688167a770" (UID: "5d685979-b511-488d-9df5-6d688167a770"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:59:21.868793 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.868764 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-istio-data" (OuterVolumeSpecName: "istio-data") pod "5d685979-b511-488d-9df5-6d688167a770" (UID: "5d685979-b511-488d-9df5-6d688167a770"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:59:21.870629 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.870602 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d685979-b511-488d-9df5-6d688167a770-istio-token" (OuterVolumeSpecName: "istio-token") pod "5d685979-b511-488d-9df5-6d688167a770" (UID: "5d685979-b511-488d-9df5-6d688167a770"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:59:21.870768 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.870655 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d685979-b511-488d-9df5-6d688167a770-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5d685979-b511-488d-9df5-6d688167a770" (UID: "5d685979-b511-488d-9df5-6d688167a770"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 23:59:21.871002 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.870979 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5d685979-b511-488d-9df5-6d688167a770" (UID: "5d685979-b511-488d-9df5-6d688167a770"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:59:21.871090 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.871033 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d685979-b511-488d-9df5-6d688167a770-kube-api-access-mx6qh" (OuterVolumeSpecName: "kube-api-access-mx6qh") pod "5d685979-b511-488d-9df5-6d688167a770" (UID: "5d685979-b511-488d-9df5-6d688167a770"). InnerVolumeSpecName "kube-api-access-mx6qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:59:21.969696 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.969665 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d685979-b511-488d-9df5-6d688167a770-istio-podinfo\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:59:21.969696 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.969691 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-credential-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:59:21.969696 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.969703 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx6qh\" (UniqueName: \"kubernetes.io/projected/5d685979-b511-488d-9df5-6d688167a770-kube-api-access-mx6qh\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:59:21.969944 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.969711 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-istio-data\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:59:21.969944 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.969733 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d685979-b511-488d-9df5-6d688167a770-istiod-ca-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:59:21.969944 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.969742 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d685979-b511-488d-9df5-6d688167a770-istio-token\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:59:21.969944 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:21.969750 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d685979-b511-488d-9df5-6d688167a770-istio-envoy\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 06 23:59:22.427974 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:22.427939 2571 generic.go:356] "Generic (PLEG): container finished" podID="5d685979-b511-488d-9df5-6d688167a770" containerID="15dc281ce1eb2d922a4b4b950633a8eee90b0062aed286eef65b09e2c75864e9" exitCode=0 Jul 06 23:59:22.428150 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:22.427994 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" Jul 06 23:59:22.428150 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:22.428019 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" event={"ID":"5d685979-b511-488d-9df5-6d688167a770","Type":"ContainerDied","Data":"15dc281ce1eb2d922a4b4b950633a8eee90b0062aed286eef65b09e2c75864e9"} Jul 06 23:59:22.428150 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:22.428105 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd" event={"ID":"5d685979-b511-488d-9df5-6d688167a770","Type":"ContainerDied","Data":"aaabd0be831e9bfd54793f33107a40ffc57ed4ec5f3537d0b142cabe867dd528"} Jul 06 23:59:22.428150 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:22.428129 2571 scope.go:117] "RemoveContainer" containerID="15dc281ce1eb2d922a4b4b950633a8eee90b0062aed286eef65b09e2c75864e9" Jul 06 23:59:22.437986 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:22.437969 2571 scope.go:117] "RemoveContainer" containerID="14935df0f57c4e2c09674693039687ab62dded246f93e7c45f13c20ad258f4c1" Jul 06 23:59:22.445969 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:22.445955 2571 scope.go:117] "RemoveContainer" containerID="15dc281ce1eb2d922a4b4b950633a8eee90b0062aed286eef65b09e2c75864e9" Jul 06 23:59:22.446223 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:59:22.446202 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15dc281ce1eb2d922a4b4b950633a8eee90b0062aed286eef65b09e2c75864e9\": container with ID starting with 15dc281ce1eb2d922a4b4b950633a8eee90b0062aed286eef65b09e2c75864e9 not found: ID does not exist" containerID="15dc281ce1eb2d922a4b4b950633a8eee90b0062aed286eef65b09e2c75864e9" Jul 06 23:59:22.446302 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:22.446235 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15dc281ce1eb2d922a4b4b950633a8eee90b0062aed286eef65b09e2c75864e9"} err="failed to get container status \"15dc281ce1eb2d922a4b4b950633a8eee90b0062aed286eef65b09e2c75864e9\": rpc error: code = NotFound desc = could not find container \"15dc281ce1eb2d922a4b4b950633a8eee90b0062aed286eef65b09e2c75864e9\": container with ID starting with 15dc281ce1eb2d922a4b4b950633a8eee90b0062aed286eef65b09e2c75864e9 not found: ID does not exist" Jul 06 23:59:22.446302 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:22.446258 2571 scope.go:117] "RemoveContainer" containerID="14935df0f57c4e2c09674693039687ab62dded246f93e7c45f13c20ad258f4c1" Jul 06 23:59:22.446496 ip-10-0-135-248 kubenswrapper[2571]: E0706 23:59:22.446474 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14935df0f57c4e2c09674693039687ab62dded246f93e7c45f13c20ad258f4c1\": container with ID starting with 14935df0f57c4e2c09674693039687ab62dded246f93e7c45f13c20ad258f4c1 not found: ID does not exist" containerID="14935df0f57c4e2c09674693039687ab62dded246f93e7c45f13c20ad258f4c1" Jul 06 23:59:22.446558 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:22.446493 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14935df0f57c4e2c09674693039687ab62dded246f93e7c45f13c20ad258f4c1"} err="failed to get container status \"14935df0f57c4e2c09674693039687ab62dded246f93e7c45f13c20ad258f4c1\": rpc error: code = NotFound desc = could not find container \"14935df0f57c4e2c09674693039687ab62dded246f93e7c45f13c20ad258f4c1\": container with ID starting with 14935df0f57c4e2c09674693039687ab62dded246f93e7c45f13c20ad258f4c1 not found: ID does not exist" Jul 06 23:59:22.449658 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:22.449637 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd"] Jul 06 23:59:22.452861 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:22.452840 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-68ab62-openshift-default-7bbcd7b88c-6w4vd"] Jul 06 23:59:23.786070 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:23.786036 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d685979-b511-488d-9df5-6d688167a770" path="/var/lib/kubelet/pods/5d685979-b511-488d-9df5-6d688167a770/volumes" Jul 06 23:59:25.433135 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.433099 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl"] Jul 06 23:59:25.433649 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.433454 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d685979-b511-488d-9df5-6d688167a770" containerName="istio-proxy" Jul 06 23:59:25.433649 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.433465 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d685979-b511-488d-9df5-6d688167a770" containerName="istio-proxy" Jul 06 23:59:25.433649 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.433474 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d685979-b511-488d-9df5-6d688167a770" containerName="istio-proxy" Jul 06 23:59:25.433649 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.433480 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d685979-b511-488d-9df5-6d688167a770" containerName="istio-proxy" Jul 06 23:59:25.433649 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.433529 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d685979-b511-488d-9df5-6d688167a770" containerName="istio-proxy" Jul 06 23:59:25.433649 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.433537 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d685979-b511-488d-9df5-6d688167a770" containerName="istio-proxy" Jul 06 23:59:25.433649 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.433543 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="839bf18d-bec2-4bfa-83f5-f8e1f4fa2b50" containerName="istio-proxy" Jul 06 23:59:25.436694 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.436666 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.439020 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.438995 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-a1e3a6-openshift-default-dockercfg-f5jqv\"" Jul 06 23:59:25.447168 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.447138 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl"] Jul 06 23:59:25.600430 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.600392 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istiod-ca-cert\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.600625 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.600461 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-token\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.600625 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.600506 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-workload-socket\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.600625 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.600530 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-workload-certs\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.600625 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.600575 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-podinfo\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.600625 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.600607 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-envoy\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.600843 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.600636 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-credential-socket\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.600843 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.600652 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-data\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.600843 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.600668 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwsf7\" (UniqueName: \"kubernetes.io/projected/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-kube-api-access-pwsf7\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.701437 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.701331 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istiod-ca-cert\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.701437 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.701386 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-token\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.701437 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.701406 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-workload-socket\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.701437 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.701429 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-workload-certs\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.701856 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.701457 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-podinfo\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.701856 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.701486 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-envoy\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.701856 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.701519 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-credential-socket\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.701856 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.701542 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-data\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.701856 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.701563 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwsf7\" (UniqueName: \"kubernetes.io/projected/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-kube-api-access-pwsf7\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.702151 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.701911 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-workload-socket\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.702151 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.701964 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-workload-certs\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.702151 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.702012 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-data\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.702295 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.702211 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istiod-ca-cert\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.702295 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.702219 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-credential-socket\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.704058 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.704037 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-envoy\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.704142 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.704094 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-podinfo\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.709904 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.709878 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-token\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.710250 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.710233 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwsf7\" (UniqueName: \"kubernetes.io/projected/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-kube-api-access-pwsf7\") pod \"e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.750199 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.750157 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:25.896454 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.896430 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl"] Jul 06 23:59:25.898542 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:59:25.898516 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8545e3f0_c35a_4408_9b64_a7a9eb3b3cce.slice/crio-3e7671d3b0ffb5d6754f836c24d15292dfc07caff6ea33d46f8f173c44f92506 WatchSource:0}: Error finding container 3e7671d3b0ffb5d6754f836c24d15292dfc07caff6ea33d46f8f173c44f92506: Status 404 returned error can't find the container with id 3e7671d3b0ffb5d6754f836c24d15292dfc07caff6ea33d46f8f173c44f92506 Jul 06 23:59:25.900871 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.900835 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:59:25.900970 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.900905 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:59:25.900970 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:25.900934 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:59:26.458410 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:26.458375 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" event={"ID":"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce","Type":"ContainerStarted","Data":"f29514621b8fffbe29bcccd26f013de4a77e0ed19e4ff431da48318033485ffa"} Jul 06 23:59:26.458850 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:26.458426 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" event={"ID":"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce","Type":"ContainerStarted","Data":"3e7671d3b0ffb5d6754f836c24d15292dfc07caff6ea33d46f8f173c44f92506"} Jul 06 23:59:26.481077 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:26.481014 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" podStartSLOduration=1.480994626 podStartE2EDuration="1.480994626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:59:26.477621865 +0000 UTC m=+2077.191676435" watchObservedRunningTime="2026-07-06 23:59:26.480994626 +0000 UTC m=+2077.195049198" Jul 06 23:59:26.750416 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:26.750326 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:26.755340 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:26.755314 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:27.463904 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:27.463871 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:27.465102 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:27.465082 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:34.405263 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:34.405226 2571 patch_prober.go:28] interesting pod/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 06 23:59:34.405697 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:34.405299 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" podUID="9ca59f7c-5f3f-4353-a44c-e923cf1842fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 06 23:59:34.504278 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:34.504244 2571 generic.go:356] "Generic (PLEG): container finished" podID="9ca59f7c-5f3f-4353-a44c-e923cf1842fe" containerID="2558ee74c959fd63885790e339631ae6ce45f1e2516e09d03504757f8d286305" exitCode=0 Jul 06 23:59:34.504461 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:34.504315 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" event={"ID":"9ca59f7c-5f3f-4353-a44c-e923cf1842fe","Type":"ContainerDied","Data":"2558ee74c959fd63885790e339631ae6ce45f1e2516e09d03504757f8d286305"} Jul 06 23:59:34.504764 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:34.504750 2571 scope.go:117] "RemoveContainer" containerID="2558ee74c959fd63885790e339631ae6ce45f1e2516e09d03504757f8d286305" Jul 06 23:59:34.505808 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:34.505778 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:59:34.505922 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:34.505862 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:59:34.505922 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:34.505911 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:59:35.512507 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:35.512459 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" event={"ID":"9ca59f7c-5f3f-4353-a44c-e923cf1842fe","Type":"ContainerStarted","Data":"f6408aafa454a431aa6c7ecc64feb4fe2ab661d5c3ff2ecd518284d9c6327169"} Jul 06 23:59:35.666144 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:35.666114 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:35.666383 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:35.666224 2571 patch_prober.go:28] interesting pod/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 06 23:59:35.666499 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:35.666481 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" podUID="9ca59f7c-5f3f-4353-a44c-e923cf1842fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 06 23:59:36.666188 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:36.666147 2571 patch_prober.go:28] interesting pod/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 06 23:59:36.666601 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:36.666212 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" podUID="9ca59f7c-5f3f-4353-a44c-e923cf1842fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 06 23:59:37.666439 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:37.666397 2571 patch_prober.go:28] interesting pod/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 06 23:59:37.666833 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:37.666465 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" podUID="9ca59f7c-5f3f-4353-a44c-e923cf1842fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 06 23:59:38.666399 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:38.666362 2571 patch_prober.go:28] interesting pod/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 06 23:59:38.666595 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:38.666426 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" podUID="9ca59f7c-5f3f-4353-a44c-e923cf1842fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 06 23:59:39.670359 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:39.670325 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:39.670858 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:39.670649 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:39.671345 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:39.671327 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 06 23:59:40.548377 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:40.548342 2571 generic.go:356] "Generic (PLEG): container finished" podID="8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" containerID="f29514621b8fffbe29bcccd26f013de4a77e0ed19e4ff431da48318033485ffa" exitCode=0 Jul 06 23:59:40.548603 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:40.548422 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" event={"ID":"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce","Type":"ContainerDied","Data":"f29514621b8fffbe29bcccd26f013de4a77e0ed19e4ff431da48318033485ffa"} Jul 06 23:59:40.548995 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:40.548977 2571 scope.go:117] "RemoveContainer" containerID="f29514621b8fffbe29bcccd26f013de4a77e0ed19e4ff431da48318033485ffa" Jul 06 23:59:40.550167 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:40.550135 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:59:40.550239 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:40.550220 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:59:40.550285 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:40.550267 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 23:59:41.556012 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:41.555975 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" event={"ID":"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce","Type":"ContainerStarted","Data":"85061710b62ea5a12753451bebcbca3fe2044aedb08318e68c750e9aae588255"} Jul 06 23:59:41.750960 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:41.750932 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:41.752120 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:41.752095 2571 patch_prober.go:28] interesting pod/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 06 23:59:41.752235 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:41.752145 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" podUID="8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 06 23:59:42.463959 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:42.463919 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:42.750600 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:42.750513 2571 patch_prober.go:28] interesting pod/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 06 23:59:42.750600 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:42.750572 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" podUID="8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 06 23:59:43.751198 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:43.751161 2571 patch_prober.go:28] interesting pod/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 06 23:59:43.751685 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:43.751233 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" podUID="8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 06 23:59:44.751407 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:44.751371 2571 patch_prober.go:28] interesting pod/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 06 23:59:44.751927 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:44.751443 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" podUID="8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 06 23:59:45.755342 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:45.755313 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:45.756367 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:45.756347 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 06 23:59:49.875517 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:49.875481 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 06 23:59:49.894773 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:49.894749 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 06 23:59:56.670152 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.670114 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq"] Jul 06 23:59:56.685903 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.685869 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq"] Jul 06 23:59:56.686076 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.685964 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.689242 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.689215 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a1e3a6\"/\"default-dockercfg-w5pz4\"" Jul 06 23:59:56.689380 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.689260 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a1e3a6\"/\"auth-test-model-a1e3a6-kserve-self-signed-certs\"" Jul 06 23:59:56.689380 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.689290 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a1e3a6\"/\"openshift-service-ca.crt\"" Jul 06 23:59:56.689380 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.689215 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a1e3a6\"/\"kube-root-ca.crt\"" Jul 06 23:59:56.804076 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.804029 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-home\") pod \"auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.804282 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.804090 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-tmp-dir\") pod \"auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.804282 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.804119 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-model-cache\") pod \"auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.804282 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.804182 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcblm\" (UniqueName: \"kubernetes.io/projected/040c0d3f-34b4-4445-9156-0475c7cb2a4e-kube-api-access-vcblm\") pod \"auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.804282 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.804245 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/040c0d3f-34b4-4445-9156-0475c7cb2a4e-tls-certs\") pod \"auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.804282 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.804268 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-dshm\") pod \"auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.905044 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.905012 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-home\") pod \"auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.905044 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.905050 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-tmp-dir\") pod \"auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.905293 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.905068 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-model-cache\") pod \"auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.905293 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.905097 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcblm\" (UniqueName: \"kubernetes.io/projected/040c0d3f-34b4-4445-9156-0475c7cb2a4e-kube-api-access-vcblm\") pod \"auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.905293 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.905143 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/040c0d3f-34b4-4445-9156-0475c7cb2a4e-tls-certs\") pod \"auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.905293 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.905161 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-dshm\") pod \"auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.905506 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.905483 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-tmp-dir\") pod \"auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.905568 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.905512 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-model-cache\") pod \"auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.905568 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.905554 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-home\") pod \"auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.907515 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.907494 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-dshm\") pod \"auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.907817 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.907801 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/040c0d3f-34b4-4445-9156-0475c7cb2a4e-tls-certs\") pod \"auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.913744 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.913690 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcblm\" (UniqueName: \"kubernetes.io/projected/040c0d3f-34b4-4445-9156-0475c7cb2a4e-kube-api-access-vcblm\") pod \"auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:56.995708 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:56.995613 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 06 23:59:57.133499 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:57.133434 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq"] Jul 06 23:59:57.137578 ip-10-0-135-248 kubenswrapper[2571]: W0706 23:59:57.137542 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod040c0d3f_34b4_4445_9156_0475c7cb2a4e.slice/crio-f1ddfd2c0aba63ad5de9d6928d1c57c72d938db676f7e967c19fa2c398e96ebb WatchSource:0}: Error finding container f1ddfd2c0aba63ad5de9d6928d1c57c72d938db676f7e967c19fa2c398e96ebb: Status 404 returned error can't find the container with id f1ddfd2c0aba63ad5de9d6928d1c57c72d938db676f7e967c19fa2c398e96ebb Jul 06 23:59:57.644897 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:57.644858 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" event={"ID":"040c0d3f-34b4-4445-9156-0475c7cb2a4e","Type":"ContainerStarted","Data":"31a64af71c58e8631e9791ef8203259014603d05145b9bcd6f5dafe577511c3c"} Jul 06 23:59:57.645099 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:57.644903 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" event={"ID":"040c0d3f-34b4-4445-9156-0475c7cb2a4e","Type":"ContainerStarted","Data":"f1ddfd2c0aba63ad5de9d6928d1c57c72d938db676f7e967c19fa2c398e96ebb"} Jul 06 23:59:57.664402 ip-10-0-135-248 kubenswrapper[2571]: I0706 23:59:57.664349 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" podStartSLOduration=1.664335376 podStartE2EDuration="1.664335376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:59:57.661318949 +0000 UTC m=+2108.375373712" watchObservedRunningTime="2026-07-06 23:59:57.664335376 +0000 UTC m=+2108.378389946" Jul 07 00:00:06.996330 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:00:06.996292 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 07 00:00:06.996854 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:00:06.996356 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 07 00:00:07.008542 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:00:07.008519 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 07 00:00:07.705809 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:00:07.705783 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 07 00:01:58.775646 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:01:58.775607 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq"] Jul 07 00:02:13.796421 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:13.796322 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" podUID="040c0d3f-34b4-4445-9156-0475c7cb2a4e" containerName="main" containerID="cri-o://31a64af71c58e8631e9791ef8203259014603d05145b9bcd6f5dafe577511c3c" gracePeriod=45 Jul 07 00:02:14.035142 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.035114 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 07 00:02:14.170779 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.170752 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-home\") pod \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " Jul 07 00:02:14.170971 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.170796 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-dshm\") pod \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " Jul 07 00:02:14.170971 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.170840 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/040c0d3f-34b4-4445-9156-0475c7cb2a4e-tls-certs\") pod \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " Jul 07 00:02:14.170971 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.170886 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcblm\" (UniqueName: \"kubernetes.io/projected/040c0d3f-34b4-4445-9156-0475c7cb2a4e-kube-api-access-vcblm\") pod \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " Jul 07 00:02:14.170971 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.170908 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-model-cache\") pod \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " Jul 07 00:02:14.170971 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.170948 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-tmp-dir\") pod \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\" (UID: \"040c0d3f-34b4-4445-9156-0475c7cb2a4e\") " Jul 07 00:02:14.171230 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.171045 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-home" (OuterVolumeSpecName: "home") pod "040c0d3f-34b4-4445-9156-0475c7cb2a4e" (UID: "040c0d3f-34b4-4445-9156-0475c7cb2a4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:02:14.171282 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.171224 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "040c0d3f-34b4-4445-9156-0475c7cb2a4e" (UID: "040c0d3f-34b4-4445-9156-0475c7cb2a4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:02:14.171364 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.171333 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "040c0d3f-34b4-4445-9156-0475c7cb2a4e" (UID: "040c0d3f-34b4-4445-9156-0475c7cb2a4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:02:14.171495 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.171341 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-home\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:02:14.171495 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.171392 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-tmp-dir\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:02:14.173346 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.173318 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/040c0d3f-34b4-4445-9156-0475c7cb2a4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "040c0d3f-34b4-4445-9156-0475c7cb2a4e" (UID: "040c0d3f-34b4-4445-9156-0475c7cb2a4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:02:14.173443 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.173365 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-dshm" (OuterVolumeSpecName: "dshm") pod "040c0d3f-34b4-4445-9156-0475c7cb2a4e" (UID: "040c0d3f-34b4-4445-9156-0475c7cb2a4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:02:14.173443 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.173414 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040c0d3f-34b4-4445-9156-0475c7cb2a4e-kube-api-access-vcblm" (OuterVolumeSpecName: "kube-api-access-vcblm") pod "040c0d3f-34b4-4445-9156-0475c7cb2a4e" (UID: "040c0d3f-34b4-4445-9156-0475c7cb2a4e"). InnerVolumeSpecName "kube-api-access-vcblm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:02:14.272615 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.272580 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-dshm\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:02:14.272615 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.272614 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/040c0d3f-34b4-4445-9156-0475c7cb2a4e-tls-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:02:14.272836 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.272629 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcblm\" (UniqueName: \"kubernetes.io/projected/040c0d3f-34b4-4445-9156-0475c7cb2a4e-kube-api-access-vcblm\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:02:14.272836 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.272640 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/040c0d3f-34b4-4445-9156-0475c7cb2a4e-model-cache\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:02:14.371223 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.371193 2571 generic.go:356] "Generic (PLEG): container finished" podID="040c0d3f-34b4-4445-9156-0475c7cb2a4e" containerID="31a64af71c58e8631e9791ef8203259014603d05145b9bcd6f5dafe577511c3c" exitCode=0 Jul 07 00:02:14.371397 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.371241 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" Jul 07 00:02:14.371397 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.371291 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" event={"ID":"040c0d3f-34b4-4445-9156-0475c7cb2a4e","Type":"ContainerDied","Data":"31a64af71c58e8631e9791ef8203259014603d05145b9bcd6f5dafe577511c3c"} Jul 07 00:02:14.371397 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.371341 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq" event={"ID":"040c0d3f-34b4-4445-9156-0475c7cb2a4e","Type":"ContainerDied","Data":"f1ddfd2c0aba63ad5de9d6928d1c57c72d938db676f7e967c19fa2c398e96ebb"} Jul 07 00:02:14.371397 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.371374 2571 scope.go:117] "RemoveContainer" containerID="31a64af71c58e8631e9791ef8203259014603d05145b9bcd6f5dafe577511c3c" Jul 07 00:02:14.381454 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.381429 2571 scope.go:117] "RemoveContainer" containerID="31a64af71c58e8631e9791ef8203259014603d05145b9bcd6f5dafe577511c3c" Jul 07 00:02:14.381764 ip-10-0-135-248 kubenswrapper[2571]: E0707 00:02:14.381735 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31a64af71c58e8631e9791ef8203259014603d05145b9bcd6f5dafe577511c3c\": container with ID starting with 31a64af71c58e8631e9791ef8203259014603d05145b9bcd6f5dafe577511c3c not found: ID does not exist" containerID="31a64af71c58e8631e9791ef8203259014603d05145b9bcd6f5dafe577511c3c" Jul 07 00:02:14.381842 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.381778 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31a64af71c58e8631e9791ef8203259014603d05145b9bcd6f5dafe577511c3c"} err="failed to get container status \"31a64af71c58e8631e9791ef8203259014603d05145b9bcd6f5dafe577511c3c\": rpc error: code = NotFound desc = could not find container \"31a64af71c58e8631e9791ef8203259014603d05145b9bcd6f5dafe577511c3c\": container with ID starting with 31a64af71c58e8631e9791ef8203259014603d05145b9bcd6f5dafe577511c3c not found: ID does not exist" Jul 07 00:02:14.394909 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.394883 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq"] Jul 07 00:02:14.398896 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:14.398863 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a1e3a6/auth-test-model-a1e3a6-kserve-bf5cc98bc-xzsxq"] Jul 07 00:02:15.786482 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:15.786439 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="040c0d3f-34b4-4445-9156-0475c7cb2a4e" path="/var/lib/kubelet/pods/040c0d3f-34b4-4445-9156-0475c7cb2a4e/volumes" Jul 07 00:02:29.257479 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.257447 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr"] Jul 07 00:02:29.257915 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.257870 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="040c0d3f-34b4-4445-9156-0475c7cb2a4e" containerName="main" Jul 07 00:02:29.257915 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.257886 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="040c0d3f-34b4-4445-9156-0475c7cb2a4e" containerName="main" Jul 07 00:02:29.257992 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.257951 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="040c0d3f-34b4-4445-9156-0475c7cb2a4e" containerName="main" Jul 07 00:02:29.261047 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.261026 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.264536 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.264513 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a1e3a6\"/\"default-dockercfg-w5pz4\"" Jul 07 00:02:29.264674 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.264519 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a1e3a6\"/\"sub-test-model-a1e3a6-kserve-self-signed-certs\"" Jul 07 00:02:29.264674 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.264557 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a1e3a6\"/\"kube-root-ca.crt\"" Jul 07 00:02:29.264674 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.264558 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a1e3a6\"/\"openshift-service-ca.crt\"" Jul 07 00:02:29.273795 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.273772 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr"] Jul 07 00:02:29.309730 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.309693 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-dshm\") pod \"sub-test-model-a1e3a6-kserve-c5596976f-rvbxr\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.309945 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.309747 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd64e10-35ca-4613-9a65-0ba68f392270-tls-certs\") pod \"sub-test-model-a1e3a6-kserve-c5596976f-rvbxr\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.309945 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.309846 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-tmp-dir\") pod \"sub-test-model-a1e3a6-kserve-c5596976f-rvbxr\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.309945 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.309867 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8jvw\" (UniqueName: \"kubernetes.io/projected/6cd64e10-35ca-4613-9a65-0ba68f392270-kube-api-access-m8jvw\") pod \"sub-test-model-a1e3a6-kserve-c5596976f-rvbxr\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.309945 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.309899 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-model-cache\") pod \"sub-test-model-a1e3a6-kserve-c5596976f-rvbxr\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.310111 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.309988 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-home\") pod \"sub-test-model-a1e3a6-kserve-c5596976f-rvbxr\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.411024 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.410988 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-dshm\") pod \"sub-test-model-a1e3a6-kserve-c5596976f-rvbxr\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.411024 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.411027 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd64e10-35ca-4613-9a65-0ba68f392270-tls-certs\") pod \"sub-test-model-a1e3a6-kserve-c5596976f-rvbxr\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.411318 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.411106 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-tmp-dir\") pod \"sub-test-model-a1e3a6-kserve-c5596976f-rvbxr\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.411318 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.411132 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8jvw\" (UniqueName: \"kubernetes.io/projected/6cd64e10-35ca-4613-9a65-0ba68f392270-kube-api-access-m8jvw\") pod \"sub-test-model-a1e3a6-kserve-c5596976f-rvbxr\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.411318 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.411167 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-model-cache\") pod \"sub-test-model-a1e3a6-kserve-c5596976f-rvbxr\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.411318 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.411211 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-home\") pod \"sub-test-model-a1e3a6-kserve-c5596976f-rvbxr\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.411560 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.411539 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-tmp-dir\") pod \"sub-test-model-a1e3a6-kserve-c5596976f-rvbxr\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.411759 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.411738 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-home\") pod \"sub-test-model-a1e3a6-kserve-c5596976f-rvbxr\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.411817 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.411752 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-model-cache\") pod \"sub-test-model-a1e3a6-kserve-c5596976f-rvbxr\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.414209 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.414187 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-dshm\") pod \"sub-test-model-a1e3a6-kserve-c5596976f-rvbxr\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.414486 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.414464 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd64e10-35ca-4613-9a65-0ba68f392270-tls-certs\") pod \"sub-test-model-a1e3a6-kserve-c5596976f-rvbxr\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.420555 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.420529 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8jvw\" (UniqueName: \"kubernetes.io/projected/6cd64e10-35ca-4613-9a65-0ba68f392270-kube-api-access-m8jvw\") pod \"sub-test-model-a1e3a6-kserve-c5596976f-rvbxr\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.571847 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.571743 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:29.708374 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:29.708340 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr"] Jul 07 00:02:29.710654 ip-10-0-135-248 kubenswrapper[2571]: W0707 00:02:29.710625 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cd64e10_35ca_4613_9a65_0ba68f392270.slice/crio-17efee97ffa3562073388277082cd1a0e72f920a6a99448918b3095ca66395fd WatchSource:0}: Error finding container 17efee97ffa3562073388277082cd1a0e72f920a6a99448918b3095ca66395fd: Status 404 returned error can't find the container with id 17efee97ffa3562073388277082cd1a0e72f920a6a99448918b3095ca66395fd Jul 07 00:02:30.460235 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:30.460201 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" event={"ID":"6cd64e10-35ca-4613-9a65-0ba68f392270","Type":"ContainerStarted","Data":"2a2e56b79b984d9dd57bdff483c84d2aa59d04deb96c3e6757c49d67ed599000"} Jul 07 00:02:30.460235 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:30.460243 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" event={"ID":"6cd64e10-35ca-4613-9a65-0ba68f392270","Type":"ContainerStarted","Data":"17efee97ffa3562073388277082cd1a0e72f920a6a99448918b3095ca66395fd"} Jul 07 00:02:30.476281 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:30.476231 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" podStartSLOduration=1.476217282 podStartE2EDuration="1.476217282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:02:30.475292217 +0000 UTC m=+2261.189346787" watchObservedRunningTime="2026-07-07 00:02:30.476217282 +0000 UTC m=+2261.190271851" Jul 07 00:02:39.572796 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:39.572763 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:39.573341 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:39.572837 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:39.585089 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:39.585063 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:02:40.524464 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:02:40.524431 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:03:12.996471 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:12.996439 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr"] Jul 07 00:03:28.017172 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.017122 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" podUID="6cd64e10-35ca-4613-9a65-0ba68f392270" containerName="main" containerID="cri-o://2a2e56b79b984d9dd57bdff483c84d2aa59d04deb96c3e6757c49d67ed599000" gracePeriod=45 Jul 07 00:03:28.270450 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.270387 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:03:28.340330 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.340303 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-tmp-dir\") pod \"6cd64e10-35ca-4613-9a65-0ba68f392270\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " Jul 07 00:03:28.340514 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.340360 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd64e10-35ca-4613-9a65-0ba68f392270-tls-certs\") pod \"6cd64e10-35ca-4613-9a65-0ba68f392270\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " Jul 07 00:03:28.340514 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.340408 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-model-cache\") pod \"6cd64e10-35ca-4613-9a65-0ba68f392270\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " Jul 07 00:03:28.340514 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.340446 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8jvw\" (UniqueName: \"kubernetes.io/projected/6cd64e10-35ca-4613-9a65-0ba68f392270-kube-api-access-m8jvw\") pod \"6cd64e10-35ca-4613-9a65-0ba68f392270\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " Jul 07 00:03:28.340514 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.340484 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-home\") pod \"6cd64e10-35ca-4613-9a65-0ba68f392270\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " Jul 07 00:03:28.340813 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.340543 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-dshm\") pod \"6cd64e10-35ca-4613-9a65-0ba68f392270\" (UID: \"6cd64e10-35ca-4613-9a65-0ba68f392270\") " Jul 07 00:03:28.340813 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.340594 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6cd64e10-35ca-4613-9a65-0ba68f392270" (UID: "6cd64e10-35ca-4613-9a65-0ba68f392270"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:03:28.340813 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.340697 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-model-cache" (OuterVolumeSpecName: "model-cache") pod "6cd64e10-35ca-4613-9a65-0ba68f392270" (UID: "6cd64e10-35ca-4613-9a65-0ba68f392270"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:03:28.340987 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.340884 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-home" (OuterVolumeSpecName: "home") pod "6cd64e10-35ca-4613-9a65-0ba68f392270" (UID: "6cd64e10-35ca-4613-9a65-0ba68f392270"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:03:28.340987 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.340966 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-home\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:03:28.340987 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.340984 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-tmp-dir\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:03:28.341308 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.340999 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-model-cache\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:03:28.342654 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.342632 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-dshm" (OuterVolumeSpecName: "dshm") pod "6cd64e10-35ca-4613-9a65-0ba68f392270" (UID: "6cd64e10-35ca-4613-9a65-0ba68f392270"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:03:28.342779 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.342752 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd64e10-35ca-4613-9a65-0ba68f392270-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cd64e10-35ca-4613-9a65-0ba68f392270" (UID: "6cd64e10-35ca-4613-9a65-0ba68f392270"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:03:28.342861 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.342835 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd64e10-35ca-4613-9a65-0ba68f392270-kube-api-access-m8jvw" (OuterVolumeSpecName: "kube-api-access-m8jvw") pod "6cd64e10-35ca-4613-9a65-0ba68f392270" (UID: "6cd64e10-35ca-4613-9a65-0ba68f392270"). InnerVolumeSpecName "kube-api-access-m8jvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:03:28.442302 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.442276 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd64e10-35ca-4613-9a65-0ba68f392270-dshm\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:03:28.442302 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.442298 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd64e10-35ca-4613-9a65-0ba68f392270-tls-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:03:28.442462 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.442309 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8jvw\" (UniqueName: \"kubernetes.io/projected/6cd64e10-35ca-4613-9a65-0ba68f392270-kube-api-access-m8jvw\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:03:28.768416 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.768381 2571 generic.go:356] "Generic (PLEG): container finished" podID="6cd64e10-35ca-4613-9a65-0ba68f392270" containerID="2a2e56b79b984d9dd57bdff483c84d2aa59d04deb96c3e6757c49d67ed599000" exitCode=0 Jul 07 00:03:28.768607 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.768436 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" Jul 07 00:03:28.768607 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.768466 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" event={"ID":"6cd64e10-35ca-4613-9a65-0ba68f392270","Type":"ContainerDied","Data":"2a2e56b79b984d9dd57bdff483c84d2aa59d04deb96c3e6757c49d67ed599000"} Jul 07 00:03:28.768607 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.768509 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr" event={"ID":"6cd64e10-35ca-4613-9a65-0ba68f392270","Type":"ContainerDied","Data":"17efee97ffa3562073388277082cd1a0e72f920a6a99448918b3095ca66395fd"} Jul 07 00:03:28.768607 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.768525 2571 scope.go:117] "RemoveContainer" containerID="2a2e56b79b984d9dd57bdff483c84d2aa59d04deb96c3e6757c49d67ed599000" Jul 07 00:03:28.778052 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.778036 2571 scope.go:117] "RemoveContainer" containerID="2a2e56b79b984d9dd57bdff483c84d2aa59d04deb96c3e6757c49d67ed599000" Jul 07 00:03:28.778326 ip-10-0-135-248 kubenswrapper[2571]: E0707 00:03:28.778309 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a2e56b79b984d9dd57bdff483c84d2aa59d04deb96c3e6757c49d67ed599000\": container with ID starting with 2a2e56b79b984d9dd57bdff483c84d2aa59d04deb96c3e6757c49d67ed599000 not found: ID does not exist" containerID="2a2e56b79b984d9dd57bdff483c84d2aa59d04deb96c3e6757c49d67ed599000" Jul 07 00:03:28.778369 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.778336 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a2e56b79b984d9dd57bdff483c84d2aa59d04deb96c3e6757c49d67ed599000"} err="failed to get container status \"2a2e56b79b984d9dd57bdff483c84d2aa59d04deb96c3e6757c49d67ed599000\": rpc error: code = NotFound desc = could not find container \"2a2e56b79b984d9dd57bdff483c84d2aa59d04deb96c3e6757c49d67ed599000\": container with ID starting with 2a2e56b79b984d9dd57bdff483c84d2aa59d04deb96c3e6757c49d67ed599000 not found: ID does not exist" Jul 07 00:03:28.788524 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.788504 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr"] Jul 07 00:03:28.791980 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:28.791962 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a1e3a6/sub-test-model-a1e3a6-kserve-c5596976f-rvbxr"] Jul 07 00:03:29.785489 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:29.785456 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cd64e10-35ca-4613-9a65-0ba68f392270" path="/var/lib/kubelet/pods/6cd64e10-35ca-4613-9a65-0ba68f392270/volumes" Jul 07 00:03:47.086950 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.086875 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk"] Jul 07 00:03:47.087379 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.087315 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd64e10-35ca-4613-9a65-0ba68f392270" containerName="main" Jul 07 00:03:47.087379 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.087331 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd64e10-35ca-4613-9a65-0ba68f392270" containerName="main" Jul 07 00:03:47.087453 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.087434 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd64e10-35ca-4613-9a65-0ba68f392270" containerName="main" Jul 07 00:03:47.091832 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.091814 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.094359 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.094333 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a1e3a6\"/\"openshift-service-ca.crt\"" Jul 07 00:03:47.095387 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.095226 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a1e3a6\"/\"rate-test-model-a1e3a6-kserve-self-signed-certs\"" Jul 07 00:03:47.095387 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.095257 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a1e3a6\"/\"kube-root-ca.crt\"" Jul 07 00:03:47.095594 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.095504 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a1e3a6\"/\"default-dockercfg-w5pz4\"" Jul 07 00:03:47.096135 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.096110 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk"] Jul 07 00:03:47.205012 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.204980 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-dshm\") pod \"rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.205211 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.205024 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-model-cache\") pod \"rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.205211 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.205044 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b69304cd-c038-4068-ba83-9dd0c8d7b22f-tls-certs\") pod \"rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.205211 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.205064 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx65d\" (UniqueName: \"kubernetes.io/projected/b69304cd-c038-4068-ba83-9dd0c8d7b22f-kube-api-access-rx65d\") pod \"rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.205380 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.205236 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-home\") pod \"rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.205380 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.205296 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-tmp-dir\") pod \"rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.306543 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.306511 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-home\") pod \"rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.306753 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.306555 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-tmp-dir\") pod \"rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.306753 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.306577 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-dshm\") pod \"rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.306753 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.306699 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-model-cache\") pod \"rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.306911 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.306772 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b69304cd-c038-4068-ba83-9dd0c8d7b22f-tls-certs\") pod \"rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.306911 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.306801 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx65d\" (UniqueName: \"kubernetes.io/projected/b69304cd-c038-4068-ba83-9dd0c8d7b22f-kube-api-access-rx65d\") pod \"rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.307081 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.306989 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-home\") pod \"rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.307134 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.307084 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-tmp-dir\") pod \"rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.307185 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.307169 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-model-cache\") pod \"rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.309044 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.309019 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-dshm\") pod \"rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.309446 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.309415 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b69304cd-c038-4068-ba83-9dd0c8d7b22f-tls-certs\") pod \"rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.315194 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.315166 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx65d\" (UniqueName: \"kubernetes.io/projected/b69304cd-c038-4068-ba83-9dd0c8d7b22f-kube-api-access-rx65d\") pod \"rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.403169 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.403076 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:47.540756 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.540702 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk"] Jul 07 00:03:47.542397 ip-10-0-135-248 kubenswrapper[2571]: W0707 00:03:47.542364 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb69304cd_c038_4068_ba83_9dd0c8d7b22f.slice/crio-035486ceb5c9060c417aa64f18a01d87b726f7f5aaa802e5aed9024abec49ba6 WatchSource:0}: Error finding container 035486ceb5c9060c417aa64f18a01d87b726f7f5aaa802e5aed9024abec49ba6: Status 404 returned error can't find the container with id 035486ceb5c9060c417aa64f18a01d87b726f7f5aaa802e5aed9024abec49ba6 Jul 07 00:03:47.866631 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.866596 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" event={"ID":"b69304cd-c038-4068-ba83-9dd0c8d7b22f","Type":"ContainerStarted","Data":"5c00f7ecf491191b28c803bed8617c032c7501152361214f5bc6260ed3cdd31d"} Jul 07 00:03:47.866631 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.866637 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" event={"ID":"b69304cd-c038-4068-ba83-9dd0c8d7b22f","Type":"ContainerStarted","Data":"035486ceb5c9060c417aa64f18a01d87b726f7f5aaa802e5aed9024abec49ba6"} Jul 07 00:03:47.883977 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:47.883917 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" podStartSLOduration=0.883896667 podStartE2EDuration="883.896667ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 00:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:03:47.881656565 +0000 UTC m=+2338.595711136" watchObservedRunningTime="2026-07-07 00:03:47.883896667 +0000 UTC m=+2338.597951238" Jul 07 00:03:57.403882 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:57.403850 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:57.404359 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:57.403906 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:57.416430 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:57.416402 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:03:57.931611 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:03:57.931583 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:04:30.063687 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:30.063651 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk"] Jul 07 00:04:45.084321 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.084287 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" podUID="b69304cd-c038-4068-ba83-9dd0c8d7b22f" containerName="main" containerID="cri-o://5c00f7ecf491191b28c803bed8617c032c7501152361214f5bc6260ed3cdd31d" gracePeriod=45 Jul 07 00:04:45.333314 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.333290 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:04:45.412919 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.412890 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-tmp-dir\") pod \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " Jul 07 00:04:45.413063 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.412956 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx65d\" (UniqueName: \"kubernetes.io/projected/b69304cd-c038-4068-ba83-9dd0c8d7b22f-kube-api-access-rx65d\") pod \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " Jul 07 00:04:45.413063 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.412974 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-dshm\") pod \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " Jul 07 00:04:45.413063 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.413008 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-model-cache\") pod \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " Jul 07 00:04:45.413063 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.413024 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b69304cd-c038-4068-ba83-9dd0c8d7b22f-tls-certs\") pod \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " Jul 07 00:04:45.413063 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.413038 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-home\") pod \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\" (UID: \"b69304cd-c038-4068-ba83-9dd0c8d7b22f\") " Jul 07 00:04:45.413296 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.413198 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b69304cd-c038-4068-ba83-9dd0c8d7b22f" (UID: "b69304cd-c038-4068-ba83-9dd0c8d7b22f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:04:45.413356 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.413309 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b69304cd-c038-4068-ba83-9dd0c8d7b22f" (UID: "b69304cd-c038-4068-ba83-9dd0c8d7b22f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:04:45.413410 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.413340 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-tmp-dir\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:04:45.413461 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.413409 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-home" (OuterVolumeSpecName: "home") pod "b69304cd-c038-4068-ba83-9dd0c8d7b22f" (UID: "b69304cd-c038-4068-ba83-9dd0c8d7b22f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:04:45.415186 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.415165 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-dshm" (OuterVolumeSpecName: "dshm") pod "b69304cd-c038-4068-ba83-9dd0c8d7b22f" (UID: "b69304cd-c038-4068-ba83-9dd0c8d7b22f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:04:45.415305 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.415284 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b69304cd-c038-4068-ba83-9dd0c8d7b22f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b69304cd-c038-4068-ba83-9dd0c8d7b22f" (UID: "b69304cd-c038-4068-ba83-9dd0c8d7b22f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 00:04:45.415364 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.415342 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69304cd-c038-4068-ba83-9dd0c8d7b22f-kube-api-access-rx65d" (OuterVolumeSpecName: "kube-api-access-rx65d") pod "b69304cd-c038-4068-ba83-9dd0c8d7b22f" (UID: "b69304cd-c038-4068-ba83-9dd0c8d7b22f"). InnerVolumeSpecName "kube-api-access-rx65d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:04:45.513923 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.513895 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx65d\" (UniqueName: \"kubernetes.io/projected/b69304cd-c038-4068-ba83-9dd0c8d7b22f-kube-api-access-rx65d\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:04:45.513923 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.513922 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-dshm\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:04:45.514078 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.513936 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-model-cache\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:04:45.514078 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.513948 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b69304cd-c038-4068-ba83-9dd0c8d7b22f-tls-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:04:45.514078 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:45.513959 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b69304cd-c038-4068-ba83-9dd0c8d7b22f-home\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:04:46.180043 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:46.180012 2571 generic.go:356] "Generic (PLEG): container finished" podID="b69304cd-c038-4068-ba83-9dd0c8d7b22f" containerID="5c00f7ecf491191b28c803bed8617c032c7501152361214f5bc6260ed3cdd31d" exitCode=0 Jul 07 00:04:46.180503 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:46.180054 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" event={"ID":"b69304cd-c038-4068-ba83-9dd0c8d7b22f","Type":"ContainerDied","Data":"5c00f7ecf491191b28c803bed8617c032c7501152361214f5bc6260ed3cdd31d"} Jul 07 00:04:46.180503 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:46.180072 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" Jul 07 00:04:46.180503 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:46.180099 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk" event={"ID":"b69304cd-c038-4068-ba83-9dd0c8d7b22f","Type":"ContainerDied","Data":"035486ceb5c9060c417aa64f18a01d87b726f7f5aaa802e5aed9024abec49ba6"} Jul 07 00:04:46.180503 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:46.180120 2571 scope.go:117] "RemoveContainer" containerID="5c00f7ecf491191b28c803bed8617c032c7501152361214f5bc6260ed3cdd31d" Jul 07 00:04:46.189130 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:46.189113 2571 scope.go:117] "RemoveContainer" containerID="5c00f7ecf491191b28c803bed8617c032c7501152361214f5bc6260ed3cdd31d" Jul 07 00:04:46.189381 ip-10-0-135-248 kubenswrapper[2571]: E0707 00:04:46.189360 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c00f7ecf491191b28c803bed8617c032c7501152361214f5bc6260ed3cdd31d\": container with ID starting with 5c00f7ecf491191b28c803bed8617c032c7501152361214f5bc6260ed3cdd31d not found: ID does not exist" containerID="5c00f7ecf491191b28c803bed8617c032c7501152361214f5bc6260ed3cdd31d" Jul 07 00:04:46.189453 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:46.189389 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c00f7ecf491191b28c803bed8617c032c7501152361214f5bc6260ed3cdd31d"} err="failed to get container status \"5c00f7ecf491191b28c803bed8617c032c7501152361214f5bc6260ed3cdd31d\": rpc error: code = NotFound desc = could not find container \"5c00f7ecf491191b28c803bed8617c032c7501152361214f5bc6260ed3cdd31d\": container with ID starting with 5c00f7ecf491191b28c803bed8617c032c7501152361214f5bc6260ed3cdd31d not found: ID does not exist" Jul 07 00:04:46.196001 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:46.195972 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk"] Jul 07 00:04:46.198104 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:46.198081 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a1e3a6/rate-test-model-a1e3a6-kserve-67d9654cf9-5vsjk"] Jul 07 00:04:47.786014 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:47.785973 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b69304cd-c038-4068-ba83-9dd0c8d7b22f" path="/var/lib/kubelet/pods/b69304cd-c038-4068-ba83-9dd0c8d7b22f/volumes" Jul 07 00:04:49.918048 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:49.918017 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 07 00:04:49.934365 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:04:49.934343 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c7gg2_23472647-b51d-4d6b-9662-3f56bade3b4d/ovn-acl-logging/0.log" Jul 07 00:05:13.569342 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:13.569260 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5"] Jul 07 00:05:13.569909 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:13.569501 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" podUID="9ca59f7c-5f3f-4353-a44c-e923cf1842fe" containerName="istio-proxy" containerID="cri-o://f6408aafa454a431aa6c7ecc64feb4fe2ab661d5c3ff2ecd518284d9c6327169" gracePeriod=30 Jul 07 00:05:18.825642 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:18.825618 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 07 00:05:18.997818 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:18.997786 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-token\") pod \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " Jul 07 00:05:18.998000 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:18.997839 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-credential-socket\") pod \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " Jul 07 00:05:18.998000 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:18.997871 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-envoy\") pod \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " Jul 07 00:05:18.998000 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:18.997886 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-workload-socket\") pod \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " Jul 07 00:05:18.998000 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:18.997906 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-podinfo\") pod \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " Jul 07 00:05:18.998000 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:18.997941 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-workload-certs\") pod \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " Jul 07 00:05:18.998000 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:18.997974 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrstw\" (UniqueName: \"kubernetes.io/projected/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-kube-api-access-jrstw\") pod \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " Jul 07 00:05:18.998322 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:18.998009 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istiod-ca-cert\") pod \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " Jul 07 00:05:18.998322 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:18.998065 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-data\") pod \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\" (UID: \"9ca59f7c-5f3f-4353-a44c-e923cf1842fe\") " Jul 07 00:05:18.998322 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:18.998222 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9ca59f7c-5f3f-4353-a44c-e923cf1842fe" (UID: "9ca59f7c-5f3f-4353-a44c-e923cf1842fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:05:18.998322 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:18.998234 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9ca59f7c-5f3f-4353-a44c-e923cf1842fe" (UID: "9ca59f7c-5f3f-4353-a44c-e923cf1842fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:05:18.998547 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:18.998381 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-credential-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:05:18.998547 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:18.998398 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-workload-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:05:18.998869 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:18.998547 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9ca59f7c-5f3f-4353-a44c-e923cf1842fe" (UID: "9ca59f7c-5f3f-4353-a44c-e923cf1842fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:05:18.998869 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:18.998534 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9ca59f7c-5f3f-4353-a44c-e923cf1842fe" (UID: "9ca59f7c-5f3f-4353-a44c-e923cf1842fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:05:18.999430 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:18.999402 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-data" (OuterVolumeSpecName: "istio-data") pod "9ca59f7c-5f3f-4353-a44c-e923cf1842fe" (UID: "9ca59f7c-5f3f-4353-a44c-e923cf1842fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:05:19.000257 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.000232 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9ca59f7c-5f3f-4353-a44c-e923cf1842fe" (UID: "9ca59f7c-5f3f-4353-a44c-e923cf1842fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:05:19.000360 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.000270 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-token" (OuterVolumeSpecName: "istio-token") pod "9ca59f7c-5f3f-4353-a44c-e923cf1842fe" (UID: "9ca59f7c-5f3f-4353-a44c-e923cf1842fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:05:19.000429 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.000401 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9ca59f7c-5f3f-4353-a44c-e923cf1842fe" (UID: "9ca59f7c-5f3f-4353-a44c-e923cf1842fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:05:19.000500 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.000474 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-kube-api-access-jrstw" (OuterVolumeSpecName: "kube-api-access-jrstw") pod "9ca59f7c-5f3f-4353-a44c-e923cf1842fe" (UID: "9ca59f7c-5f3f-4353-a44c-e923cf1842fe"). InnerVolumeSpecName "kube-api-access-jrstw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:05:19.099848 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.099817 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-envoy\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:05:19.099848 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.099843 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-podinfo\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:05:19.099848 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.099853 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-workload-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:05:19.100081 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.099863 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrstw\" (UniqueName: \"kubernetes.io/projected/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-kube-api-access-jrstw\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:05:19.100081 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.099872 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istiod-ca-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:05:19.100081 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.099880 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-data\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:05:19.100081 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.099888 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ca59f7c-5f3f-4353-a44c-e923cf1842fe-istio-token\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:05:19.357071 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.356978 2571 generic.go:356] "Generic (PLEG): container finished" podID="9ca59f7c-5f3f-4353-a44c-e923cf1842fe" containerID="f6408aafa454a431aa6c7ecc64feb4fe2ab661d5c3ff2ecd518284d9c6327169" exitCode=0 Jul 07 00:05:19.357071 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.357029 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" Jul 07 00:05:19.357071 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.357056 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" event={"ID":"9ca59f7c-5f3f-4353-a44c-e923cf1842fe","Type":"ContainerDied","Data":"f6408aafa454a431aa6c7ecc64feb4fe2ab661d5c3ff2ecd518284d9c6327169"} Jul 07 00:05:19.357342 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.357107 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5" event={"ID":"9ca59f7c-5f3f-4353-a44c-e923cf1842fe","Type":"ContainerDied","Data":"80921957cf04bdc191afa138d0e530cb9096a96a88bf8bfb57a023390af28412"} Jul 07 00:05:19.357342 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.357130 2571 scope.go:117] "RemoveContainer" containerID="f6408aafa454a431aa6c7ecc64feb4fe2ab661d5c3ff2ecd518284d9c6327169" Jul 07 00:05:19.366598 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.366583 2571 scope.go:117] "RemoveContainer" containerID="2558ee74c959fd63885790e339631ae6ce45f1e2516e09d03504757f8d286305" Jul 07 00:05:19.375039 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.375022 2571 scope.go:117] "RemoveContainer" containerID="f6408aafa454a431aa6c7ecc64feb4fe2ab661d5c3ff2ecd518284d9c6327169" Jul 07 00:05:19.375290 ip-10-0-135-248 kubenswrapper[2571]: E0707 00:05:19.375273 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6408aafa454a431aa6c7ecc64feb4fe2ab661d5c3ff2ecd518284d9c6327169\": container with ID starting with f6408aafa454a431aa6c7ecc64feb4fe2ab661d5c3ff2ecd518284d9c6327169 not found: ID does not exist" containerID="f6408aafa454a431aa6c7ecc64feb4fe2ab661d5c3ff2ecd518284d9c6327169" Jul 07 00:05:19.375359 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.375302 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6408aafa454a431aa6c7ecc64feb4fe2ab661d5c3ff2ecd518284d9c6327169"} err="failed to get container status \"f6408aafa454a431aa6c7ecc64feb4fe2ab661d5c3ff2ecd518284d9c6327169\": rpc error: code = NotFound desc = could not find container \"f6408aafa454a431aa6c7ecc64feb4fe2ab661d5c3ff2ecd518284d9c6327169\": container with ID starting with f6408aafa454a431aa6c7ecc64feb4fe2ab661d5c3ff2ecd518284d9c6327169 not found: ID does not exist" Jul 07 00:05:19.375359 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.375327 2571 scope.go:117] "RemoveContainer" containerID="2558ee74c959fd63885790e339631ae6ce45f1e2516e09d03504757f8d286305" Jul 07 00:05:19.375564 ip-10-0-135-248 kubenswrapper[2571]: E0707 00:05:19.375548 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2558ee74c959fd63885790e339631ae6ce45f1e2516e09d03504757f8d286305\": container with ID starting with 2558ee74c959fd63885790e339631ae6ce45f1e2516e09d03504757f8d286305 not found: ID does not exist" containerID="2558ee74c959fd63885790e339631ae6ce45f1e2516e09d03504757f8d286305" Jul 07 00:05:19.375604 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.375571 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2558ee74c959fd63885790e339631ae6ce45f1e2516e09d03504757f8d286305"} err="failed to get container status \"2558ee74c959fd63885790e339631ae6ce45f1e2516e09d03504757f8d286305\": rpc error: code = NotFound desc = could not find container \"2558ee74c959fd63885790e339631ae6ce45f1e2516e09d03504757f8d286305\": container with ID starting with 2558ee74c959fd63885790e339631ae6ce45f1e2516e09d03504757f8d286305 not found: ID does not exist" Jul 07 00:05:19.378873 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.378851 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5"] Jul 07 00:05:19.384375 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.384358 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-9ea6e9-openshift-default-7dfc57dd47-stgv5"] Jul 07 00:05:19.785842 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:19.785804 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ca59f7c-5f3f-4353-a44c-e923cf1842fe" path="/var/lib/kubelet/pods/9ca59f7c-5f3f-4353-a44c-e923cf1842fe/volumes" Jul 07 00:05:20.634120 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:20.634087 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl"] Jul 07 00:05:20.634593 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:20.634344 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" podUID="8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" containerName="istio-proxy" containerID="cri-o://85061710b62ea5a12753451bebcbca3fe2044aedb08318e68c750e9aae588255" gracePeriod=30 Jul 07 00:05:21.320665 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:21.320630 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85f77bb897-8zv69_bf0205d9-de1a-4043-b6a4-68576017e352/authorino/0.log" Jul 07 00:05:25.884829 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.884803 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 07 00:05:25.957121 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.957088 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istiod-ca-cert\") pod \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " Jul 07 00:05:25.957121 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.957122 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-workload-socket\") pod \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " Jul 07 00:05:25.957364 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.957148 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwsf7\" (UniqueName: \"kubernetes.io/projected/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-kube-api-access-pwsf7\") pod \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " Jul 07 00:05:25.957364 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.957178 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-credential-socket\") pod \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " Jul 07 00:05:25.957364 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.957240 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-envoy\") pod \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " Jul 07 00:05:25.957364 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.957289 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-workload-certs\") pod \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " Jul 07 00:05:25.957364 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.957325 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-token\") pod \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " Jul 07 00:05:25.957606 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.957359 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" (UID: "8545e3f0-c35a-4408-9b64-a7a9eb3b3cce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:05:25.957606 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.957382 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-podinfo\") pod \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " Jul 07 00:05:25.957606 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.957521 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" (UID: "8545e3f0-c35a-4408-9b64-a7a9eb3b3cce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:05:25.957606 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.957531 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" (UID: "8545e3f0-c35a-4408-9b64-a7a9eb3b3cce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 00:05:25.957606 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.957564 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" (UID: "8545e3f0-c35a-4408-9b64-a7a9eb3b3cce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:05:25.957880 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.957669 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-workload-certs\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:05:25.957880 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.957686 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istiod-ca-cert\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:05:25.957880 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.957698 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-workload-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:05:25.957880 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.957711 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-credential-socket\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:05:25.959475 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.959451 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-kube-api-access-pwsf7" (OuterVolumeSpecName: "kube-api-access-pwsf7") pod "8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" (UID: "8545e3f0-c35a-4408-9b64-a7a9eb3b3cce"). InnerVolumeSpecName "kube-api-access-pwsf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:05:25.959796 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.959778 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" (UID: "8545e3f0-c35a-4408-9b64-a7a9eb3b3cce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 00:05:25.959908 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.959893 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" (UID: "8545e3f0-c35a-4408-9b64-a7a9eb3b3cce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:05:25.959959 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:25.959902 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-token" (OuterVolumeSpecName: "istio-token") pod "8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" (UID: "8545e3f0-c35a-4408-9b64-a7a9eb3b3cce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 00:05:26.058075 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.058047 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-data\") pod \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\" (UID: \"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce\") " Jul 07 00:05:26.058211 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.058202 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwsf7\" (UniqueName: \"kubernetes.io/projected/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-kube-api-access-pwsf7\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:05:26.058266 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.058214 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-envoy\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:05:26.058266 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.058223 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-token\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:05:26.058266 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.058231 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-podinfo\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:05:26.058638 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.058618 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-data" (OuterVolumeSpecName: "istio-data") pod "8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" (UID: "8545e3f0-c35a-4408-9b64-a7a9eb3b3cce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 00:05:26.158602 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.158564 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce-istio-data\") on node \"ip-10-0-135-248.ec2.internal\" DevicePath \"\"" Jul 07 00:05:26.395940 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.395853 2571 generic.go:356] "Generic (PLEG): container finished" podID="8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" containerID="85061710b62ea5a12753451bebcbca3fe2044aedb08318e68c750e9aae588255" exitCode=0 Jul 07 00:05:26.395940 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.395903 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" Jul 07 00:05:26.396112 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.395935 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" event={"ID":"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce","Type":"ContainerDied","Data":"85061710b62ea5a12753451bebcbca3fe2044aedb08318e68c750e9aae588255"} Jul 07 00:05:26.396112 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.395986 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl" event={"ID":"8545e3f0-c35a-4408-9b64-a7a9eb3b3cce","Type":"ContainerDied","Data":"3e7671d3b0ffb5d6754f836c24d15292dfc07caff6ea33d46f8f173c44f92506"} Jul 07 00:05:26.396112 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.396010 2571 scope.go:117] "RemoveContainer" containerID="85061710b62ea5a12753451bebcbca3fe2044aedb08318e68c750e9aae588255" Jul 07 00:05:26.406174 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.406156 2571 scope.go:117] "RemoveContainer" containerID="f29514621b8fffbe29bcccd26f013de4a77e0ed19e4ff431da48318033485ffa" Jul 07 00:05:26.415002 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.414984 2571 scope.go:117] "RemoveContainer" containerID="85061710b62ea5a12753451bebcbca3fe2044aedb08318e68c750e9aae588255" Jul 07 00:05:26.415287 ip-10-0-135-248 kubenswrapper[2571]: E0707 00:05:26.415255 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85061710b62ea5a12753451bebcbca3fe2044aedb08318e68c750e9aae588255\": container with ID starting with 85061710b62ea5a12753451bebcbca3fe2044aedb08318e68c750e9aae588255 not found: ID does not exist" containerID="85061710b62ea5a12753451bebcbca3fe2044aedb08318e68c750e9aae588255" Jul 07 00:05:26.415366 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.415299 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85061710b62ea5a12753451bebcbca3fe2044aedb08318e68c750e9aae588255"} err="failed to get container status \"85061710b62ea5a12753451bebcbca3fe2044aedb08318e68c750e9aae588255\": rpc error: code = NotFound desc = could not find container \"85061710b62ea5a12753451bebcbca3fe2044aedb08318e68c750e9aae588255\": container with ID starting with 85061710b62ea5a12753451bebcbca3fe2044aedb08318e68c750e9aae588255 not found: ID does not exist" Jul 07 00:05:26.415366 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.415324 2571 scope.go:117] "RemoveContainer" containerID="f29514621b8fffbe29bcccd26f013de4a77e0ed19e4ff431da48318033485ffa" Jul 07 00:05:26.415542 ip-10-0-135-248 kubenswrapper[2571]: E0707 00:05:26.415525 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f29514621b8fffbe29bcccd26f013de4a77e0ed19e4ff431da48318033485ffa\": container with ID starting with f29514621b8fffbe29bcccd26f013de4a77e0ed19e4ff431da48318033485ffa not found: ID does not exist" containerID="f29514621b8fffbe29bcccd26f013de4a77e0ed19e4ff431da48318033485ffa" Jul 07 00:05:26.415593 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.415550 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f29514621b8fffbe29bcccd26f013de4a77e0ed19e4ff431da48318033485ffa"} err="failed to get container status \"f29514621b8fffbe29bcccd26f013de4a77e0ed19e4ff431da48318033485ffa\": rpc error: code = NotFound desc = could not find container \"f29514621b8fffbe29bcccd26f013de4a77e0ed19e4ff431da48318033485ffa\": container with ID starting with f29514621b8fffbe29bcccd26f013de4a77e0ed19e4ff431da48318033485ffa not found: ID does not exist" Jul 07 00:05:26.420021 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.419996 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl"] Jul 07 00:05:26.423309 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.423286 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-a1e3a6-openshift-default-844db6784c-bdbbl"] Jul 07 00:05:26.439171 ip-10-0-135-248 kubenswrapper[2571]: E0707 00:05:26.439141 2571 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8545e3f0_c35a_4408_9b64_a7a9eb3b3cce.slice/crio-3e7671d3b0ffb5d6754f836c24d15292dfc07caff6ea33d46f8f173c44f92506\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8545e3f0_c35a_4408_9b64_a7a9eb3b3cce.slice\": RecentStats: unable to find data in memory cache]" Jul 07 00:05:26.935614 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:26.935583 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-585888c566-5s5m7_80f2e998-37f7-4e22-a3c2-e4158bd937e8/manager/0.log" Jul 07 00:05:27.575081 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:27.575056 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d555ff84d-wq97k_2a628d66-50b8-4d07-a2c9-3143e8a02ceb/copy-manifests/0.log" Jul 07 00:05:27.588406 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:27.588382 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d555ff84d-wq97k_2a628d66-50b8-4d07-a2c9-3143e8a02ceb/manager/0.log" Jul 07 00:05:27.785780 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:27.785751 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" path="/var/lib/kubelet/pods/8545e3f0-c35a-4408-9b64-a7a9eb3b3cce/volumes" Jul 07 00:05:28.585775 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:28.585748 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx_1bae9384-1e9a-4da5-abb0-7b2d61d4732c/util/0.log" Jul 07 00:05:28.593513 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:28.593492 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx_1bae9384-1e9a-4da5-abb0-7b2d61d4732c/pull/0.log" Jul 07 00:05:28.602151 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:28.602136 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx_1bae9384-1e9a-4da5-abb0-7b2d61d4732c/extract/0.log" Jul 07 00:05:28.720173 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:28.720149 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m_71e387a9-a244-4335-9fdf-38c8c24cdbbf/util/0.log" Jul 07 00:05:28.726899 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:28.726877 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m_71e387a9-a244-4335-9fdf-38c8c24cdbbf/pull/0.log" Jul 07 00:05:28.733537 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:28.733514 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m_71e387a9-a244-4335-9fdf-38c8c24cdbbf/extract/0.log" Jul 07 00:05:28.854391 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:28.854326 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q_eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8/util/0.log" Jul 07 00:05:28.861364 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:28.861340 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q_eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8/pull/0.log" Jul 07 00:05:28.867595 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:28.867577 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q_eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8/extract/0.log" Jul 07 00:05:28.987417 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:28.987395 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5_7320a85d-c52d-4f7a-bfa8-467cfa332ed3/util/0.log" Jul 07 00:05:28.993441 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:28.993417 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5_7320a85d-c52d-4f7a-bfa8-467cfa332ed3/pull/0.log" Jul 07 00:05:28.999925 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:28.999909 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5_7320a85d-c52d-4f7a-bfa8-467cfa332ed3/extract/0.log" Jul 07 00:05:29.127697 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:29.127637 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85f77bb897-8zv69_bf0205d9-de1a-4043-b6a4-68576017e352/authorino/0.log" Jul 07 00:05:29.376979 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:29.376960 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-dzxcf_28f9c1ca-5ec6-4f14-aaba-66dfb0f8fe0c/manager/0.log" Jul 07 00:05:29.624489 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:29.624464 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-g8qcs_e74cb8a1-cf4d-474a-9cb1-5cb378dd680e/registry-server/0.log" Jul 07 00:05:29.752538 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:29.752511 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-rb7gf_33306557-1799-46c7-897d-ceae9e249f79/manager/0.log" Jul 07 00:05:30.019032 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:30.018988 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-t44rq_ebe5f1bf-a1cc-4a28-a938-9bc1a11c4e4e/manager/0.log" Jul 07 00:05:30.430870 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:30.430844 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf48kwjn_89db98c1-0628-4d66-a459-c80ad509c607/istio-proxy/0.log" Jul 07 00:05:30.932321 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:30.932292 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-24flx_d0c72526-90d2-46aa-ac1f-4bd0625ba248/istio-proxy/1.log" Jul 07 00:05:31.059553 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:31.059527 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-6ph7p_d948f999-2d98-4c3e-92fa-19cef82771ae/payload-pre-processing/0.log" Jul 07 00:05:31.683136 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:31.683111 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-6455c_d8c0c1a2-a64c-4430-9fc6-4091f680d85b/main/0.log" Jul 07 00:05:32.057605 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:32.057520 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzplnq_eda0239d-0167-45e7-947c-5f0167a0dfe6/main/0.log" Jul 07 00:05:32.313949 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:32.313866 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-q6ml7_216e2e88-850e-4ea7-aaea-e47e44a425ff/main/0.log" Jul 07 00:05:39.958905 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:39.958870 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rsnfg_a58729bb-f5a9-4bce-8cfc-3f98f6dd2401/global-pull-secret-syncer/0.log" Jul 07 00:05:40.081758 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:40.081712 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jdmhg_c92b54e3-3f51-45c9-96a7-92f26052db71/konnectivity-agent/0.log" Jul 07 00:05:40.101754 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:40.101710 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-248.ec2.internal_7916f8702abca842d06320c25e14c3c0/haproxy/0.log" Jul 07 00:05:44.209603 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:44.209571 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx_1bae9384-1e9a-4da5-abb0-7b2d61d4732c/extract/0.log" Jul 07 00:05:44.236009 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:44.235980 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx_1bae9384-1e9a-4da5-abb0-7b2d61d4732c/util/0.log" Jul 07 00:05:44.264299 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:44.264278 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592sptx_1bae9384-1e9a-4da5-abb0-7b2d61d4732c/pull/0.log" Jul 07 00:05:44.291729 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:44.291694 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m_71e387a9-a244-4335-9fdf-38c8c24cdbbf/extract/0.log" Jul 07 00:05:44.316575 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:44.316551 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m_71e387a9-a244-4335-9fdf-38c8c24cdbbf/util/0.log" Jul 07 00:05:44.339585 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:44.339567 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zh72m_71e387a9-a244-4335-9fdf-38c8c24cdbbf/pull/0.log" Jul 07 00:05:44.375769 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:44.375748 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q_eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8/extract/0.log" Jul 07 00:05:44.401098 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:44.401076 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q_eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8/util/0.log" Jul 07 00:05:44.430616 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:44.430602 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wxs5q_eccc2e39-66b4-4cd3-93e9-97fdbdfc41f8/pull/0.log" Jul 07 00:05:44.462108 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:44.462056 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5_7320a85d-c52d-4f7a-bfa8-467cfa332ed3/extract/0.log" Jul 07 00:05:44.488629 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:44.488612 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5_7320a85d-c52d-4f7a-bfa8-467cfa332ed3/util/0.log" Jul 07 00:05:44.511358 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:44.511340 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef145sk5_7320a85d-c52d-4f7a-bfa8-467cfa332ed3/pull/0.log" Jul 07 00:05:44.775971 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:44.775887 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85f77bb897-8zv69_bf0205d9-de1a-4043-b6a4-68576017e352/authorino/0.log" Jul 07 00:05:44.897500 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:44.897471 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-dzxcf_28f9c1ca-5ec6-4f14-aaba-66dfb0f8fe0c/manager/0.log" Jul 07 00:05:44.968610 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:44.968583 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-g8qcs_e74cb8a1-cf4d-474a-9cb1-5cb378dd680e/registry-server/0.log" Jul 07 00:05:45.115894 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:45.115816 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-rb7gf_33306557-1799-46c7-897d-ceae9e249f79/manager/0.log" Jul 07 00:05:45.265090 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:45.265058 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-t44rq_ebe5f1bf-a1cc-4a28-a938-9bc1a11c4e4e/manager/0.log" Jul 07 00:05:47.108645 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:47.108612 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-wxmwj_ae132236-cbea-4271-b751-535163747780/kube-state-metrics/0.log" Jul 07 00:05:47.131899 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:47.131875 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-wxmwj_ae132236-cbea-4271-b751-535163747780/kube-rbac-proxy-main/0.log" Jul 07 00:05:47.156960 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:47.156921 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-wxmwj_ae132236-cbea-4271-b751-535163747780/kube-rbac-proxy-self/0.log" Jul 07 00:05:47.244438 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:47.244401 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-kr2lr_fd890264-dc09-4401-ab1a-6d3b37fb5bde/monitoring-plugin/0.log" Jul 07 00:05:47.463534 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:47.463507 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hfr7m_8195eebd-4c62-4a63-8d35-d05df15f75d9/node-exporter/0.log" Jul 07 00:05:47.487511 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:47.487485 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hfr7m_8195eebd-4c62-4a63-8d35-d05df15f75d9/kube-rbac-proxy/0.log" Jul 07 00:05:47.515910 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:47.515885 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hfr7m_8195eebd-4c62-4a63-8d35-d05df15f75d9/init-textfile/0.log" Jul 07 00:05:50.345827 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.345744 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl"] Jul 07 00:05:50.346256 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.346208 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" containerName="istio-proxy" Jul 07 00:05:50.346256 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.346225 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" containerName="istio-proxy" Jul 07 00:05:50.346256 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.346253 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b69304cd-c038-4068-ba83-9dd0c8d7b22f" containerName="main" Jul 07 00:05:50.346418 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.346264 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b69304cd-c038-4068-ba83-9dd0c8d7b22f" containerName="main" Jul 07 00:05:50.346418 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.346285 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ca59f7c-5f3f-4353-a44c-e923cf1842fe" containerName="istio-proxy" Jul 07 00:05:50.346418 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.346294 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ca59f7c-5f3f-4353-a44c-e923cf1842fe" containerName="istio-proxy" Jul 07 00:05:50.346418 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.346309 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ca59f7c-5f3f-4353-a44c-e923cf1842fe" containerName="istio-proxy" Jul 07 00:05:50.346418 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.346318 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ca59f7c-5f3f-4353-a44c-e923cf1842fe" containerName="istio-proxy" Jul 07 00:05:50.347030 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.347003 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ca59f7c-5f3f-4353-a44c-e923cf1842fe" containerName="istio-proxy" Jul 07 00:05:50.347220 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.347206 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b69304cd-c038-4068-ba83-9dd0c8d7b22f" containerName="main" Jul 07 00:05:50.347320 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.347300 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" containerName="istio-proxy" Jul 07 00:05:50.347320 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.347321 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8545e3f0-c35a-4408-9b64-a7a9eb3b3cce" containerName="istio-proxy" Jul 07 00:05:50.356100 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.356078 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:50.358645 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.358624 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ttmnp\"/\"openshift-service-ca.crt\"" Jul 07 00:05:50.358776 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.358646 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ttmnp\"/\"default-dockercfg-dngpf\"" Jul 07 00:05:50.359536 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.359520 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ttmnp\"/\"kube-root-ca.crt\"" Jul 07 00:05:50.360152 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.360132 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl"] Jul 07 00:05:50.432564 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.432539 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kcj2\" (UniqueName: \"kubernetes.io/projected/e965b9ea-bc88-436c-892c-f54c656ff48b-kube-api-access-2kcj2\") pod \"perf-node-gather-daemonset-vqmfl\" (UID: \"e965b9ea-bc88-436c-892c-f54c656ff48b\") " pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:50.432665 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.432588 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e965b9ea-bc88-436c-892c-f54c656ff48b-proc\") pod \"perf-node-gather-daemonset-vqmfl\" (UID: \"e965b9ea-bc88-436c-892c-f54c656ff48b\") " pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:50.432665 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.432643 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e965b9ea-bc88-436c-892c-f54c656ff48b-podres\") pod \"perf-node-gather-daemonset-vqmfl\" (UID: \"e965b9ea-bc88-436c-892c-f54c656ff48b\") " pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:50.432782 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.432679 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e965b9ea-bc88-436c-892c-f54c656ff48b-lib-modules\") pod \"perf-node-gather-daemonset-vqmfl\" (UID: \"e965b9ea-bc88-436c-892c-f54c656ff48b\") " pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:50.432782 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.432735 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e965b9ea-bc88-436c-892c-f54c656ff48b-sys\") pod \"perf-node-gather-daemonset-vqmfl\" (UID: \"e965b9ea-bc88-436c-892c-f54c656ff48b\") " pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:50.533392 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.533362 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e965b9ea-bc88-436c-892c-f54c656ff48b-podres\") pod \"perf-node-gather-daemonset-vqmfl\" (UID: \"e965b9ea-bc88-436c-892c-f54c656ff48b\") " pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:50.533506 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.533407 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e965b9ea-bc88-436c-892c-f54c656ff48b-lib-modules\") pod \"perf-node-gather-daemonset-vqmfl\" (UID: \"e965b9ea-bc88-436c-892c-f54c656ff48b\") " pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:50.533506 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.533437 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e965b9ea-bc88-436c-892c-f54c656ff48b-sys\") pod \"perf-node-gather-daemonset-vqmfl\" (UID: \"e965b9ea-bc88-436c-892c-f54c656ff48b\") " pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:50.533506 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.533479 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kcj2\" (UniqueName: \"kubernetes.io/projected/e965b9ea-bc88-436c-892c-f54c656ff48b-kube-api-access-2kcj2\") pod \"perf-node-gather-daemonset-vqmfl\" (UID: \"e965b9ea-bc88-436c-892c-f54c656ff48b\") " pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:50.533666 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.533524 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e965b9ea-bc88-436c-892c-f54c656ff48b-lib-modules\") pod \"perf-node-gather-daemonset-vqmfl\" (UID: \"e965b9ea-bc88-436c-892c-f54c656ff48b\") " pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:50.533666 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.533527 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e965b9ea-bc88-436c-892c-f54c656ff48b-podres\") pod \"perf-node-gather-daemonset-vqmfl\" (UID: \"e965b9ea-bc88-436c-892c-f54c656ff48b\") " pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:50.533666 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.533554 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e965b9ea-bc88-436c-892c-f54c656ff48b-sys\") pod \"perf-node-gather-daemonset-vqmfl\" (UID: \"e965b9ea-bc88-436c-892c-f54c656ff48b\") " pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:50.533666 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.533547 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e965b9ea-bc88-436c-892c-f54c656ff48b-proc\") pod \"perf-node-gather-daemonset-vqmfl\" (UID: \"e965b9ea-bc88-436c-892c-f54c656ff48b\") " pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:50.533666 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.533595 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e965b9ea-bc88-436c-892c-f54c656ff48b-proc\") pod \"perf-node-gather-daemonset-vqmfl\" (UID: \"e965b9ea-bc88-436c-892c-f54c656ff48b\") " pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:50.541188 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.541164 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kcj2\" (UniqueName: \"kubernetes.io/projected/e965b9ea-bc88-436c-892c-f54c656ff48b-kube-api-access-2kcj2\") pod \"perf-node-gather-daemonset-vqmfl\" (UID: \"e965b9ea-bc88-436c-892c-f54c656ff48b\") " pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:50.666979 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.666959 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:50.798938 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:50.798913 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl"] Jul 07 00:05:50.800441 ip-10-0-135-248 kubenswrapper[2571]: W0707 00:05:50.800404 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode965b9ea_bc88_436c_892c_f54c656ff48b.slice/crio-bdd703c6413675463ea6dcbd3309aad1835802caa2f30d05d7a74fa7d5b3cc63 WatchSource:0}: Error finding container bdd703c6413675463ea6dcbd3309aad1835802caa2f30d05d7a74fa7d5b3cc63: Status 404 returned error can't find the container with id bdd703c6413675463ea6dcbd3309aad1835802caa2f30d05d7a74fa7d5b3cc63 Jul 07 00:05:51.529863 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:51.529830 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" event={"ID":"e965b9ea-bc88-436c-892c-f54c656ff48b","Type":"ContainerStarted","Data":"7aebc05fbcc6e806a754db4ae7fceded0391fbae15b4e93f2754ea3fe33b6332"} Jul 07 00:05:51.530283 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:51.529880 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" event={"ID":"e965b9ea-bc88-436c-892c-f54c656ff48b","Type":"ContainerStarted","Data":"bdd703c6413675463ea6dcbd3309aad1835802caa2f30d05d7a74fa7d5b3cc63"} Jul 07 00:05:51.530283 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:51.529985 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:51.546691 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:51.546647 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" podStartSLOduration=1.5466359029999999 podStartE2EDuration="1.546635903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 00:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 00:05:51.544063559 +0000 UTC m=+2462.258118131" watchObservedRunningTime="2026-07-07 00:05:51.546635903 +0000 UTC m=+2462.260690473" Jul 07 00:05:52.226884 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:52.226853 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-58kg2_1a157e85-8951-46be-bbef-ea3583fc73aa/dns/0.log" Jul 07 00:05:52.248383 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:52.248360 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-58kg2_1a157e85-8951-46be-bbef-ea3583fc73aa/kube-rbac-proxy/0.log" Jul 07 00:05:52.431795 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:52.431770 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l6dhh_e746a856-21ba-4c28-9545-6e6d6e299f49/dns-node-resolver/0.log" Jul 07 00:05:53.039615 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:53.039580 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kpwsm_17ddd37f-3cef-44d2-990a-1ca5e4b484fc/node-ca/0.log" Jul 07 00:05:54.027497 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:54.027471 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf48kwjn_89db98c1-0628-4d66-a459-c80ad509c607/istio-proxy/0.log" Jul 07 00:05:54.354942 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:54.354853 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-24flx_d0c72526-90d2-46aa-ac1f-4bd0625ba248/istio-proxy/0.log" Jul 07 00:05:54.381800 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:54.381778 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-24flx_d0c72526-90d2-46aa-ac1f-4bd0625ba248/istio-proxy/1.log" Jul 07 00:05:54.417915 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:54.417889 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-6ph7p_d948f999-2d98-4c3e-92fa-19cef82771ae/payload-pre-processing/0.log" Jul 07 00:05:55.215628 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:55.215603 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b4s7d_a06f7512-2cd6-402b-a4eb-c85052df5dbd/serve-healthcheck-canary/0.log" Jul 07 00:05:55.713114 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:55.713084 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-frnkj_2c162094-42a9-4902-a4ea-0b8ab848b05d/kube-rbac-proxy/0.log" Jul 07 00:05:55.735924 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:55.735902 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-frnkj_2c162094-42a9-4902-a4ea-0b8ab848b05d/exporter/0.log" Jul 07 00:05:55.758601 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:55.758578 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-frnkj_2c162094-42a9-4902-a4ea-0b8ab848b05d/extractor/0.log" Jul 07 00:05:57.544851 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:57.544821 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ttmnp/perf-node-gather-daemonset-vqmfl" Jul 07 00:05:58.533886 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:58.533856 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-585888c566-5s5m7_80f2e998-37f7-4e22-a3c2-e4158bd937e8/manager/0.log" Jul 07 00:05:58.827924 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:58.827836 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d555ff84d-wq97k_2a628d66-50b8-4d07-a2c9-3143e8a02ceb/manager/0.log" Jul 07 00:05:58.845312 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:05:58.845290 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d555ff84d-wq97k_2a628d66-50b8-4d07-a2c9-3143e8a02ceb/copy-manifests/0.log" Jul 07 00:06:00.417330 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:00.417295 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7ff7c895fb-7tbp2_7d5b93ea-f4b1-4b73-a362-98a6ac4cf6dd/manager/0.log" Jul 07 00:06:00.699118 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:00.699092 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf48kwjn_89db98c1-0628-4d66-a459-c80ad509c607/istio-proxy/0.log" Jul 07 00:06:00.919893 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:00.919864 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-24flx_d0c72526-90d2-46aa-ac1f-4bd0625ba248/istio-proxy/0.log" Jul 07 00:06:00.951776 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:00.951695 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-24flx_d0c72526-90d2-46aa-ac1f-4bd0625ba248/istio-proxy/1.log" Jul 07 00:06:00.971467 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:00.971446 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-6ph7p_d948f999-2d98-4c3e-92fa-19cef82771ae/payload-pre-processing/0.log" Jul 07 00:06:02.183995 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:02.183969 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf48kwjn_89db98c1-0628-4d66-a459-c80ad509c607/istio-proxy/0.log" Jul 07 00:06:02.436595 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:02.436515 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-24flx_d0c72526-90d2-46aa-ac1f-4bd0625ba248/istio-proxy/0.log" Jul 07 00:06:02.462174 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:02.462145 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-24flx_d0c72526-90d2-46aa-ac1f-4bd0625ba248/istio-proxy/1.log" Jul 07 00:06:02.483828 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:02.483807 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-6ph7p_d948f999-2d98-4c3e-92fa-19cef82771ae/payload-pre-processing/0.log" Jul 07 00:06:05.679608 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:05.679580 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-rkhp5_e4edfb48-79ac-4ffa-a279-96c5aa1a8348/migrator/0.log" Jul 07 00:06:05.700271 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:05.700245 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-rkhp5_e4edfb48-79ac-4ffa-a279-96c5aa1a8348/graceful-termination/0.log" Jul 07 00:06:07.281841 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:07.281813 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bvl5t_f1332668-fa67-4d07-9a0a-4975eadc0623/kube-multus-additional-cni-plugins/0.log" Jul 07 00:06:07.303299 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:07.303270 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bvl5t_f1332668-fa67-4d07-9a0a-4975eadc0623/egress-router-binary-copy/0.log" Jul 07 00:06:07.324459 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:07.324437 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bvl5t_f1332668-fa67-4d07-9a0a-4975eadc0623/cni-plugins/0.log" Jul 07 00:06:07.346984 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:07.346962 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bvl5t_f1332668-fa67-4d07-9a0a-4975eadc0623/bond-cni-plugin/0.log" Jul 07 00:06:07.368400 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:07.368383 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bvl5t_f1332668-fa67-4d07-9a0a-4975eadc0623/routeoverride-cni/0.log" Jul 07 00:06:07.389936 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:07.389916 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bvl5t_f1332668-fa67-4d07-9a0a-4975eadc0623/whereabouts-cni-bincopy/0.log" Jul 07 00:06:07.410607 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:07.410591 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bvl5t_f1332668-fa67-4d07-9a0a-4975eadc0623/whereabouts-cni/0.log" Jul 07 00:06:07.619491 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:07.619415 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bgdhp_3d4a0f6b-bcbe-4399-b6b4-f31c9f96b1f5/kube-multus/0.log" Jul 07 00:06:07.815846 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:07.815815 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mdnxn_f9260140-57c2-4913-adc3-7497f54cd33b/network-metrics-daemon/0.log" Jul 07 00:06:07.837794 ip-10-0-135-248 kubenswrapper[2571]: I0707 00:06:07.837772 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mdnxn_f9260140-57c2-4913-adc3-7497f54cd33b/kube-rbac-proxy/0.log"